It is known as The Kasbah and for 100 years, until the mid-1980s, it was a thriving area where thousands of men and women worked.
But then came the so-called Cod Wars between Britain and Iceland, followed by the adoption of the EU Common Fisheries Policy, and Grimsby fell into a steep decline.
Nowadays only a handful of trawlers call the town their home port and, although it remains a major fish market and the UK's biggest seafood processing town, the Kasbah and the surrounding streets are facing oblivion.
Grimsby ranks in the lowest percentile when it comes to deprivation and has become a byword for industrial decline. In 2015 the town was up in arms at its portrayal by Sacha Baron Cohen in the comedy, Grimsby.
But there is hope at hand.
In October last year North East Lincolnshire Council granted it Conservation Area status and plans are afoot by Associated British Ports (ABP) — who own the majority of the land — to renovate the area.
'Fantastic News for Grimsby'
"This is fantastic news for Grimsby. The Kasbah is a very special place that has huge potential for regeneration," said Louise Brennan, Planning Director, East Midlands at Historic England at the time.
"Across the country, developers and businesses are repairing and reusing historic buildings, turning local history into economic success. We believe that making the Kasbah into a Conservation Area will help make this happen in Grimsby," she added.
But with Brexit due to come into force in March 2019, Grimsby will not be able to call on the huge amounts of European Union regeneration grants which have breathed life into other UK towns.
Grimsby Voted for Brexit
In the June 2016 referendum around 70 percent of people in Grimsby voted for Brexit.
Nick Triplow, the author of Distant Water: Stories from Grimsby's Fishing Fleet, explained why the area is so anti-European.
"There remains a hefty UKIP vote and there was a big majority for Brexit. The context for that is that many people still blame the EU for the demise of the fishing industry," Mr. Triplow told Sputnik.
Earlier this year former UKIP leader Nigel Farage and other campaigners dumped fish in the Thames near the Houses of Parliament in protest at Theresa May's inability to free Britain from the Common Fisheries Policy during the Brexit transition period.
But Mr. Triplow said it was unrealistic to think the British fishing industry could be salvaged after Brexit.
'Pipedream' To Expect UK Fishing Fleet to Return
"Right up until the mid-1980s Grimsby was a huge fishing port but it's a pipedream to think it could be resurrected. There are no fish stocks anyway and it would depend on huge investment. Where's that going to come from?" Mr. Triplow told Sputnik.
He said there had been many different schemes over the years to breathe new life into the Kasbah area but each had come to nought.
"The land is owned by ABP so you need their consent to do anything but previous regeneration bids were always scuppered by them. They were simply more focused on the new car dock and bringing in timber and they weren't engaged with the regeneration process at that time," Mr. Triplow told Sputnik.
But Dafydd Williams, head of communications and corporate affairs with ABP Humber, said ABP Grimsby's port had a "major role to play" in the town's future.
"The Kasbah is a great example of this as the historic buildings have the potential to attract new jobs and growth for the local area with the right investment and support. ABP is delighted to be working closely with North East Lincs Council and Historic England to help to make better use of the Kasbah whilst preserving its heritage for future generations. We are looking forward to working side by side with the local council in the hope of bringing new business across to the region," Mr. Williams told Sputnik.
An ABP spokesperson said there was a common misconception that to have a business on the port estate in Grimsby, companies had to be related to the maritime industry.
"We haven't specifically explored EU regeneration money at this stage, but we are open to any sources of funding that will help to deliver our ambitions for the future of Grimsby," the ABP spokesperson told Sputnik.
A spokesman for North East Lincolnshire Council said they had drafted a Greater Grimsby Town Deal, which included ambitious plans to grow the local economy by more than £216 million per year.
They hoped to build 7,700 homes and create 5,400 jobs and they said key to this was "a new higher education presence, waterfront housing, cultural and leisure facilities through the redevelopment of historic sites including the Silo at Victoria Mills and the Kasbah Conservation Area at the Port of Grimsby."
All fine words, but the people of Grimsby have heard them all before.
"Whether there would have been European money if it weren't for Brexit is a moot point but there was an opportunity to do it in the early 2000s and ABP weren't engaged. You still need a whole lot of private funding," Mr. Triplow told Sputnik.
The views of the speaker do not necessarily reflect those of Sputnik.