Novel forms of data visualization (work in progress)

Hi all,

This is post where I experiment with novel forms of data visualization using ggplot and grid (under the R programming language.) The file is still a work in progress, and I don’t provide a lot in the way of explanations. The important idea in this project is that we’re trying to create a way of showing all the available data while simultaneously making the conclusion of said data intuitive to read.

I’ll save a complete explanation as to why I’m doing this for when it’s done. What I will say now is that expanding the set of people who can make use of the vast amount of information we have available, may be one of the best things we can do to improve our collective intelligence and capabilities.

poly-grid_data-vis

What I’m learning about AI

I recently spent some time going through “Neural Networks from Scratch” where a guy goes over how to build an AI without any of the additional tools at your disposal. I’ve learned a lot about what building neural networks looks like. And I find that learning about it has in some sense lead to more fascination, while at the same time demystifying it to the point where I can understand why some people are totally unimpressed with what AI can do.

Rather than go over the menutia of it. I think it’s informative to reflect on the general vibe of what I’m seeing.

Essentially, whats happening is the computer is creating a group of matrices to multiply together in clever ways that do something useful. That is my assessment of what AI is. once you get past how back propogration works, and once you get past how matrix multiplication works. the rest of it is just finding ways of massaging data to fit your needs. I think it’s, in it’s own way sort of fanscinating how such a simple process can lead to such substantial results.

Having said that, “massaging the data” is a process that takes years of research to figure out how to do right. In reading the book, I learned about some of the finding from that research. And what I mean by massaging is that you will take something that a relatively basic mathematical operation, and layer them over each other so that the parameters are still set properly. But I always believe that thinking about this is terms of principles is going to take you further than, like I said, getting into the menutia.

lets take the simplest neural network I can think of. You are a being sitting on a square with one square next to it that contains food. You’re goal is to get the food. You essentially have one choice, move in the only direction available towards the food. That’s a pretty straight forward propositions.

But let’s imagine for a second that you have no concept of what you’re doing. all you know is that getting food is good, and what you’re able to do. Let’s say you have the option to either stay in one place or move in any direction available, and you choose one of those options randomly. you might get lucky and decide to move, thereby, receiving the reward. but that cannot be assumed. and in this universe, you have no concept that there’s something next you. You can’t see, hear, or feel anything around you. the only perception you have is that you’ve made it to the food, and that that is good.

Now, lets complicate it a bit you are a dot on a 3 dot wide line. And your goal is to find the electronic food located somewhere else on the line in as few moves as possible. Your only options are to move left, or right. In this example, you essentially have 2 choices. Almost. if we include staying still as a option, and if we include moving up or down as an option, then you have 5, on your first move. Remember that if you make a “move,” you have no concept that the move you made did anything. So if you’re in the center of a 1×3 grid, and you decide to move up, you will have no concept that you haven’t moved. all you can do it make a decision at random for what to do, and count the number of times you move before you gain the food, then change how you move next time to accommedate that.

I’ve been going with the most rudimentary examples I can think of for the sake of this exercise, but you could keep uilding up uinder you get to something that resembles intelligence. Moving this example up to a 10×10 grid, or change the grid each time, and changing the location of the food based on some unknown rule, or based on no rule at all. simple doing it randomly. And then creating decisions based on, not what movements to make, but how to decide which movements to make. let’s say we modify thw rules so that you can see the squares around you and you can remember what moves you made in the past. or perhaps you given a set of hints as to where the food is. Once the parameter’s reach a certain level and the hints reach a certain level of complexity, then you start getting into the area of chatGPT.

But the fascinating thing about this for me is that while these programs are able to do these almost magical things, they do so on such simple principles taken to a certain extreme. there’s a conversation that could be had about how this idea that insignificant ideas ca be spread to greaater heights can be applied to other field. the basic principles of capitalism, for example are mostly very simple.

City Size and Quality

[Over the course of this post I pontificate about things that I only have partial knowledge of. I think what I have to say is valuable because it creates thoughts that can be tested. However, equally valuable is this disclaimer.]

One thing that has always perplexed me is the lack or correlation between the size of a given city and how well it is managed. To illustrate why this perplexes me, if a business does a bad job attracting customers, then that business will go under. If you then extend that interpretation to include cities, and think of the residents of that city as the city’s “customers”, then you would think that the cities that do the best job managing its resources and population, would have the highest population, and vice versa. But that doesn’t seem to be the case. Millions live in LA, despite it’s bad air quality, traffic, and a homelessness crisis. Millions more flock to NYC, despite its lackluster Subway quality, even worse traffic (by some measures), and (a personal pet peeve) a lack of public restrooms.

You might think that my lamentations of NYC and LA are unfair and inaccurate. There are certainly things that LA and NYC do right. And you might think that the correlation instead runs negative. Meaning that large cities are harder to manage, so they are inherently more likely to have more problems. That seems to be a general way people think about where these problems of cities come from. I disagree with this interpretation because if you look at one of the world’s largest cities, Tokyo, you see the opposite thing happening. Tokyo is both very large, and very low in crime, having a robust and successful economy (relatively speaking), and boasting one of the worlds best transit systems. (Side note: it was my first time going to Tokyo that got me interested in urban design to begin with.) And you can also find examples of very small cities and towns with a very poor and destitute population. It just doesn’t make sense. And I’m not sure what the actual answer is, but I have a few hypotheses.

It’s possible that this is merely a statistical phenomenon. The same thing can be observed in colleges. At one point the Bill and Melinda Gates Foundation at one point believed that smaller colleges were more successful than larger ones. (Citation: hearsay.) However, upon further research of the data, the true reason was found. It turns out that this simply has to do with the fact that there are more smaller colleges than larger ones. And so, simply by random chance, you are going to find more good colleges that are small than that are large. If you were to fill a bucket with 110 colleges, with their rating of “goodness”. And let’s say 10 of them are large, 100 small. You could have a situation where larger colleges are more likely to be successful, but you are still able to find more examples of successful small colleges. Let’s say 40% of large colleges are “good”, and 20% of small colleges. Then you would end up with a situation where you are able to find 4 examples of good large colleges, and 20 examples of good small colleges. And come to the conclusion that small colleges are better. Because 20 is larger than 4. (A similar situation can be observed in measuring the effectiveness of the COVID-19 vaccine. In places with a particularly high rate of vaccination, you end with more breakthrough cases than unvaccinated cases. That does not prove the vaccine is ineffective. But I digress.) And those 20 good colleges don’t even have to be particularly good. Simply by random chance, smaller colleges are more likely to admit and graduate successful students because they have fewer opportunities for bad students to mess up the average. This is true for the same reason that if you take a jar of jellybeans, the smaller the handful, the more likely you are to take more of the same color. And the larger the handful, the more likely you are to grab a distribution of colors that represents the jar as whole. So if I were to blindly grab 3 jellybeans from the jar, and all 3 of them are red, that wouldn’t prove that I’m really good at grabbing red jellybeans and really bad at grabbing purple ones. The same thing could be true for colleges, cities, companies, etc.

This type of phenomenon could explain some of the desire for small townness in popular culture. Larger cities have more opportunities to experience issues with homelessness, because there are more people, and more people means more homeless people, for example. And the same thought process could be true for other aspects of city management. Personally, I don’t buy this explanation. It does somewhat explain why there are a wide variety of socioeconomic situations in small settlements, but it doesn’t stack against my Tokyo counterexample. Barring even that, it does beg the question as to why these small successful cities don’t inevitably grow into large successful cities. (The same question arises with colleges, but this post is already running long.)

Here we get to some of the structural changes that might be impeding the success of various regions. I’ve been reading a book called, “Cities and Regions as Self Organizing Systems” by Peter Allen, and they do touch on this topic. I’ll be going over my take on it plus some additional perspectives. And in my opinion none of these explanations provide a perfectly complete picture of what is going on, but they create hypotheses that can be tested and built on.

To start with, the area that encompasses the official boundary of a given city is usually much smaller than the metropolitan area we normally associate with it. I live (at the time of writing) in Madison, WI. The population of Madison is about 250,000 if you only include the city proper, but about 500,000 if you include the surrounding suburbs. If you look at a map of the city’s limits you would see one of the most jagged areas imaginable. You could easily mistake it for a gerrymandered district. Filling the gaps is an amalgamation of towns that have either refused to integrate themselves into the city of Madison, Or have otherwise made some kind of deal with the city. Whether or not they have done this, it is often the case that property owners (in Madison and elsewhere in America) will fight tooth and nail against high density development in their neighborhood. Whether it be by forming their own section of a given city that doesn’t need to abide by the same rules, their own township, or simply protesting at council hearings. Meaning that even within one of these given “cities,” it is hard to expand on the population because (though theoretically possible to build) there is no more housing to put people in. So, without the ability to expand outward, and without the ability to expand inward and upward, the success of a city can be measured by how effectively they can segregate or shoe away “bad” residents and attract “good” residents. Which is easier to do if your town is small. But it does create this dichotomy of winners and losers in the economic game of city quality.

But this does beg the question in my mind. Surely there are at least some examples of these cities successfully integrating the less well off and/or creating a high density environment that is desirable to the wealthy. You could say that Manhattan is an example of the latter. And it’s not like every non-affluent place is also a terrible place to be. However, this explanation doesn’t quite do it for me. In the case of Manhattan, there’s a little bit of a chicken and egg problem. NYC has always been a large city compared to it’s American counterparts. This is despite many of the issues that it has faced throughout its history. It seems more likely to me that property values on the island skyrocketed for other reasons (more on that in a bit) and got it to the point where the property values would remain high regardless of how much development took place. And it’s not like the city is fully in favor of development. It’s learned ways to compromise on these things, with setback requirements, limits on developing historical landmarks, small commercial spaces at the base of their skyscrapers, and things like that that make the area attractive (and expensive) for residents and tourists alike, driving up property values. So the marginally higher population of NYC can partially be explained by their marginally better handling of high density. However, given the existence of similarly large populations with completely different ways of dealing with this issue. I don’t think it tells the whole story. And if that were the case, how come we don’t see those aforementioned small towns growing into amazing metropolises?

All in all, this is a pretty weak argument in my opinion. Worth exploring because it reveals other issues with the way that cities are designed in America. But it doesn’t say anything about the consistent rule of city size distribution. What I just destined is a contemporaneous problem that mostly takes place in America. Other parts of the world do not necessarily have quite the same attitude about urban planning. At other times in history America didn’t have the same attitude about urban planning. Yet this rule has remained true throughout region and time. We cannot look at contemporary explanations to find our answer. And we shall search, instead for things that are more consistent.

It’s possible that part of the answer lies in the common explanation for why cities form in the first place: location. Cities are (mostly) not artificial entities. They come out of the existence of settlements of individuals exploiting resources, strategic positions along trading routes, etc.. And that seems to say more about a city’s size than anything else. And this is where our modelers in “Cities and Regions as Self Organizing Systems” come in. They created simplified simulations of population dynamics. The mathematics they use is a little bit over my head, but what I can tell you is that these simulations tested differing population distributions based on random events. Trying to take into consideration factors such as the natural carrying capacity, transportation costs, innovation that then increases the carrying capacity of a given location, and so on. One thing they discovered, perhaps also not surprisingly, is that there are several stable positions that can come out of the distribution of a population. If you run the simulation several times, you will get several arrangements that will then remain consistent indefinitely. This demonstrates, perhaps, that there are some things (that may or may not be random and unknowable) that can happen early in urban development and have a lasting effect for generations.

This is observed in the real world. The proportion of the largest cities in the world has stayed consistent with itself for most of human history. (Barring instances where new lands are discovered. Such as America.) This model suggests that it was not necessarily inevitable that London would be the largest city in Europe. That if you were to run history again and allow the butterfly effect to take hold, then some other city might have just as well taken that spot and kept it. Perhaps it could have just as easily have been Rotterdam or Southampton to become the largest city in Europe instead of London. Both are located strategically. You might argue that London is located on the Thames, and that is what makes it so large. Then I ask you Why London is not located further up the Thames where it is more centrally accessible to the rest of the nation like Paris is. Conversely, why isn’t Paris located closer to the shore? You might say that this is because they are both capital cities. But it is not inevitable that the capital is the largest city in the country. The answer might just be random chance.

This consolidation of location is further exacerbated by the existence of infrastructure feeding into and out of these population centers. As well as maintaining them. Once a highway is built to Chicago, it becomes difficult to convince businesses to move their cargo into another city where it will be harder to get to. Once you have the infrastructure to feed cargo, you need to build infrastructure for the water and sewage system, this attracts more businesses, which attracts more people, and the cycle continues.

However, if a municipality fails to maintain the infrastructure necessary to maintain the population, then in theory, over a long enough period of time, an equally strategic location could snatch at the opportunity to start to build its own infrastructure. So maybe the answer is that how well managed a city is DOES have an effect on it’s population, just on a multigenerational timescale. And perhaps the issue of randomness is what keeps the proportion of the largest cities the way that they are. If you imagine large cities and small cities on an infinite timescale, some of the small cities could, in theory, overtake the larger ones eventually. But just as randomness causes them to be good in the first place. Randomness will cause them to be bad in the future, while the large cities will remain steady, with location and infrastructure to back it.

Earlier in this essay I said that the distribution of the largest cities has mostly been consistent over the course of history. Mostly. There was a time when Tenochtitlan of the Aztec empire was one of the largest cities in the world before the Spanish arrived. That empire has fallen, and the city has come with it. Maybe there is something to be said for that. Maybe over the course of generations, there is some significance to the size of the city and its ability to maintain its population. But this all needs further research.

Ranked Choice bill Reform

When you are in an environment where large groups of individuals have equal or similar decision making power, it becomes difficult and time consuming to come to any kind of consensus. Especially if all the individuals have competing interests. Even if all you need is a small majority. But there might still be a way.

You could have a system where every individual has a chance to make a proposal for how to move forward, and then rank their preference of their proposal along with everyone else’s, with logical modifiers. And by logical modifiers I mean that they would have the ability to say you want to use they’re proposal &/OR someone else’s. This system has some organizational use, but it’s most obvious application is legislative work.

For example, let’s say you (the legislature) have an infrastructure project you want to get through. Let’s call this hypothetical project A1. And you know another senator that wants a different infrastructure project. Let’s call it C22. You care more about your project than your fellow senators, and you also care about reducing the deficit. And infrastructure projects cost money. Therefore you would rather that only your plan is passed over both of your plans passing. But you would also rather both of your plans passing over nothing happening. On top of that you have another fellow senator, let’s just make up a name for him and call him Hans. He wants to fund a project to build an alien death laser to put into orbit. Let’s call this proposal MegatronDeathRay. While all of you agree that this third proposal is very important, again, you care about deficit spending. So you’re ranked choice goes as follows:
A1
A1 & C22
A1 & MegatronDeathRay
A1 & C22 & MegatronDeathRay
[Nay]
C22
MegatronDeathRay
C22 & MegatronDeathRay

Here, Nay means that you would rather that nothing is passed. Now let’s say you’re indifferent between C22 and MegatronDeathRay. Your choices could look something like this:
A1
A1 & (C22 OR MegatronDeathRay)
A1 & C22 & MegatronDeathRay
[Nay]
C22 OR MegatronDeathRay
C22 & MegatronDeathRay

If you wanted to, you could set up a way to publish your preferences without actually voting on them. By seeing the preferences of the other members, you know more exactly how to approach the issue, who you need to convince, etc.. And perhaps the biggest advantage is that it allows smaller parties or even lowly individuals to get a word in on what bills should or shouldn’t be passed. Without leading to gridlock.

Ranked choice voting has many different dynamics without the logical modifiers. And adding them in adds a whole new level of complexity. Exploring this might be a better topic for another time. One thing that is yet to be determined is what to do with the Nay vote. Initially, I figured that the end of the preference list was the nay vote. But I can still see the merits of having a way of saying, “I don’t want anything to pass, but if something does pass, make it this.” It seems like increasing granularity would be a worthwhile goal.

Why am I picking “Adaptability” as my skill to master?

I‘ve spent a majority of my life trying to resolve a mismatch of my ambitions and my practicality. When you have such a wide variety of interests that span across multiple disciplines, from art to science and psychology, and it comes time to pick a career path, you’re left with a couple options.

(1) pick out one of your interests and hope that, on a knife’s edge, you manage to pick the one that will satisfy you. This is often the life advice that people give. And there’s some merit to it. Passion is not something that you have or don’t have. It’s something you cultivate. I have experienced working at a job or doing something that I initially had no interest in to discover something that I liked about it. Or even if I didn’t, over time I grew to love certain aspects of it. I built a partnership with my coworkers. I gained a sense of pride from what I could accomplish, even if the cause held no merit to me. But I’ve also had the opposite happen to me. Sometimes at the same time. Doing something that drained me over time and held little to no hold on me.

Or (2) try to do everything and most likely crash and burn in the process. Multitasking is feasible, but difficult, unproductive, and potentially destructive. I could come up with example after example of how this works, but that might be another post.

Left with these two options, one might think that for some people (me) there is no good path to go down. And maybe there isn’t. But there’s a third option to try, which is to pick a skill (or a couple of skills) that span across all your interests. And for me, I think adaptability is that very skill. Required for anyone that wants to be innovative.

Innovation is a tricky subject. It might not be completely obvious how one would go about being more innovative over time. Perhaps it is something that is almost easy to do when you are young. But as you get old, being innovative becomes far more difficult. You’ve seen the ways of the past and you know what works. It is therefore tempting to fall into the trap of continuing the cycle of that which worked in the past in perpetuity. Unable to see the potential of new ideas and technology and they interact with the world they work in. Being young(ish), But still old enough to be able to observe this pattern in myself. I want to plan ahead for how I’m going to continue to be innovative as the forces of the conventional ways of doing things become stronger. If only there was some way I could continue to be adaptable. Do

There isn’t a college major for adaptability. And it’s hard to find ways of studying it directly. But I have a couple ideas for how I can learn this skill indirectly. For example; have you ever heard the term, “when you’re a hammer, everything looks like a nail?” That term is usually used as a negative, but I think I can use it to my advantage. By finding things, and using whatever information I can to learn how to be more adaptable.

I’m sure there are ways to learn it directly, and I will pursue those as well. Which is the subject of a later blog post. But I think there’s value in finding ways to search outside and find things that you can bring home. As I’m writing this, I am in the genesis of going to school to study Data Science. And I’m doing that for much the same reason I’m studying adaptability.

I’ve got a fascination with how large sets of data interact with the systems that govern our lives. I think understanding these systems is a major opportunity for innovation. And being able to take advantage of this opportunity is going to require being able to operate in a dynamic environment. To both see the big picture and the smaller parts and how they interact. Working in an environment where you are crossing disciplines.

I could also learn about psychology. Understand how the mind operates in different environments and how to manipulate it. I might learn more about design. Not just on a superficial level. But understand the way design affects us, and what it means for our world and our actions. And on and on…

One could come up with an infinite list of things that, in some way, relate back to an ability to maintain adaptability. I have no doubt there will be a lot of duds. And I think I’m fine with that. Not because I like duds, but because if there’s one thing that life has taught me. It’s that it’s better to try and fail than to never try at all.

Whatever path I go down. There is no way that being more adaptable is going to hurt me along the way. There may be other things that I can do that would be more effective, but it does not therefore mean that I should wait for the perfect opportunity for action. And whatever action I do take in the future. Being adaptable will bring me closer to it.