(04 = Fourth, 20 = 20th, 2 = SE, i.e., the 2nd corner counting clockwise from NE = 1)
I thought it might be fun just to pick a corner at random, look at the photograph, and see what there is to learn about its history, or at least whatever odd facts may turn up about it, via no more than a few relatively simple internet searches.
The first thing, as already mentioned, is the change of name from Fourth Avenue to Park Avenue, starting in 1860. Actually this renaming proceeded in stages, and once "Park" Avenue was established north of Grand Central Station, the stretch of it between 17th and 42nd Streets came to be known as Park Avenue South.
20th Street, of course, is straighforwardly on the "grid" proposed by the Commissioner's plan of 1811 — the famous map itself is dated 1807 — just a few blocks above Union Place at the multi-block long intersection of Bowery and the Bloomingdale Road (between 10th and 17th Streets — now, in part, Union Square) and the Parade Ground (between 23rd and 34th Streets from Third Avenue west to Seventh Avenue), later cut back in stages to the present Madison Square (the cutbacks had already begun when John Randel published his 1821 Manhattan map and were finished when the park (with its present boundaries) was opened in 1847).
The signage on the lamp post on the southeast corner of Park Avenue South and 20th Street tells us also that the avenue at this point is also known as Theodore Roosevelt Way, owing to the 26th President's birthplace being half a block to the west, at 28 East 20th Street, between Park Avenue South and Broadway.
In the other direction, mile or so to the east, 20th Street terminates at what some people call "the East 20th Street Beach", though that's a bit of a stretch as well as getting pretty far afield from Park Avenue South.
Closer by, in fact just a block away, also to the east, we have Gramercy Park, a private park as well as a neighborhood rich in history and in riches too. Gramercy Park — the neighborhood — dates from 1831, when a developer, Samuel B. Ruggles, bought Gramercy Farm from James Duane (Duane Street, downtown), a descendant of Peter Stuyvesant. The park was fenced in 1833.
The neighborhood — that is, the neighborhood of Park Avenue South and 20th Street — also has allegiances to the Flatiron District to north and west, and to Union Square to the south and west.
Not as much, this, as with some corners in the city — though there's no doubt much more one could learn — but not nothing either: history in Manhattan is almost if not more densely packed on the island than the streets themselves, and if you're not standing on a corner of great historical significance, you're probably not more than a block or two from one that is.
Doubters are directed as usual to White & Willensky's AIA Guide to New York City, in this instance to the sections "Ladies Mile" (pp. 189–200) and "Union Square (pp. 201–208).