Exploring the Climate Resilience Data Platform: A Survey of Its Assets
This post explores the Climate Resilience Data Platform, showcasing how it sources climate articles, tracks social conversations, and classifies narratives. A behind-the-scenes look at refining data products to reveal societal insights and advance the mission of RepublicOfData.io.
Over the past months, I’ve iterated quickly to create a platform that captures climate-related conversations on social networks. I shared my journey and turned a blind eye to my accumulated tech debt. Or didn’t blink twice when confronted with suspicious data quality.
The last months might not have led to any breakthrough innovation worth reporting on, but they have been busy: refactoring the asset management system, enforcing stricter typed datasets, productionizing AI agents, and more.
I think it’s now worth taking a step back and examining the platform's data assets. We won’t dive deep into any story yet (I’ll save that for the next post). Instead, we’ll survey the material we now have.
This will be a deep dive into much metadata, so buckle up!
👨💻
At RepublicOfData.io, we openly share code and journey. Here’s a list of repositories that are part of this project:
In recent days the president-elect has called for asserting U.S. control over the Panama Canal and Greenland, showing that his “America First” philosophy has an expansionist dimension.
David E. Sanger and Lisa Friedman
United States International Relations,United States Politics and Government,Trump, Donald J,Greenland,Panama Canal and Canal Zone,Presidential Election of 2024,Global Warming,Metals and Minerals,Ships and Shipping,Egede, Mute B,Arctic Regions,Denmark,Panama
In recent days the president-elect has called for asserting U.S. control over the Panama Canal and Greenland, showing that his “America First” philosophy has an expansionist dimension.
David E. Sanger and Lisa Friedman
United States International Relations,United States Politics and Government,Trump, Donald J,Greenland,Panama Canal and Canal Zone,Presidential Election of 2024,Global Warming,Metals and Minerals,Ships and Shipping,Egede, Mute B,Arctic Regions,Denmark,Panama
In recent days the president-elect has called for asserting U.S. control over the Panama Canal and Greenland, showing that his “America First” philosophy has an expansionist dimension.
David E. Sanger and Lisa Friedman
United States International Relations,United States Politics and Government,Trump, Donald J,Greenland,Panama Canal and Canal Zone,Presidential Election of 2024,Global Warming,Metals and Minerals,Ships and Shipping,Egede, Mute B,Arctic Regions,Denmark,Panama
In recent days the president-elect has called for asserting U.S. control over the Panama Canal and Greenland, showing that his “America First” philosophy has an expansionist dimension.
David E. Sanger and Lisa Friedman
United States International Relations,United States Politics and Government,Trump, Donald J,Greenland,Panama Canal and Canal Zone,Presidential Election of 2024,Global Warming,Metals and Minerals,Ships and Shipping,Egede, Mute B,Arctic Regions,Denmark,Panama
In recent days the president-elect has called for asserting U.S. control over the Panama Canal and Greenland, showing that his “America First” philosophy has an expansionist dimension.
David E. Sanger and Lisa Friedman
United States International Relations,United States Politics and Government,Trump, Donald J,Greenland,Panama Canal and Canal Zone,Presidential Election of 2024,Global Warming,Metals and Minerals,Ships and Shipping,Egede, Mute B,Arctic Regions,Denmark,Panama
Articles are the foundation of the platform. Once climate-related articles are pulled, the platform monitors social networks and capture conversations that refer to them.
Social Networks Assets
We then monitor conversations on social networks (for now, only X) that refer to the articles sourced.
Let’s now get some details on conversations that occur on X.
pretty sure it's the homelessness above all https:
1034174877488570370
jhv85
Brooklyn, NY
Writer/researcher specializing in American political development, political economy, party systems and ideology, social democracy. Columnist @compactmag_
We then continue monitoring those conversations for 24 hours to capture all their posts. Here are some details on that asset:
Some metrics:
Let’s get some posts:
TWEET_ID
TWEET_CREATED_AT
TWEET_CONVERSATION_ID
TWEET_TEXT
AUTHOR_ID
AUTHOR_USERNAME
AUTHOR_LOCATION
AUTHOR_DESCRIPTION
AUTHOR_CREATED_AT
PARTITION_HOUR_UTC_TS
RECORD_LOADING_TS
1871177534823383120
2024-12-23 07:54:23
1871176675834417432
@Matthuber78 They are just anti-progress. Unless t
199430225
HibbertMatthew
Tavistock, Devon
Renegade-redhead soixante-sixard. Broadly Liberal but I like boundaries. Blog a bit.
2010-10-06 17:09:24
2024-12-23 07:00:00
2024-12-23 13:21:37
1871178666358501848
2024-12-23 07:58:53
1871176675834417432
@collectifission Regardless of how you model the f
352833079
Matthuber78
Syracuse, NY
Geographer, Lifeblood (2013) @UMinnPress, Climate Change as Class War (2022) @VersoBooks https://t.co/OgdpkbYLz3
2011-08-11 00:08:33
2024-12-23 07:00:00
2024-12-23 13:21:37
1871182808481476922
2024-12-23 08:15:21
1871176675834417432
@Matthuber78 Grinding up olivine is possibly the b
1483186552587132932
collectifission
All about energy and what that means for people, in relation with the rest of the world. With technology we can all live well, with room for nature to flourish.
2022-01-17 16:17:24
2024-12-23 07:00:00
2024-12-23 13:21:37
1871177492033372210
2024-12-23 07:54:13
1871176675834417432
@Matthuber78 Honestly though: I know the IPCC mode
1483186552587132932
collectifission
All about energy and what that means for people, in relation with the rest of the world. With technology we can all live well, with room for nature to flourish.
2022-01-17 16:17:24
2024-12-23 07:00:00
2024-12-23 13:21:37
1871297366130934184
2024-12-23 15:50:33
1871287900257808710
@mzjacobson @nytimes Carbon capture can reduce emi
1574113788344664064
ClimateSageO
Amman
تحويل سياسة المناخ إلى ممارسة #محارب_المناخ
2022-09-25 15:09:08
2024-12-23 13:00:00
2024-12-23 19:21:43
Finally, we are geolocating the users that are part of those conversations. Some details:
And some metrics:
Let’s get some geolocated user records:
SOCIAL_NETWORK_PROFILE_ID
SOCIAL_NETWORK_PROFILE_USERNAME
LOCATION_ORDER
LOCATION
COUNTRYNAME
COUNTRYCODE
ADMINNAME1
ADMINCODE1
LATITUDE
LONGITUDE
GEOLOCATION_TS
PARTITION_HOUR_UTC_TS
1478403674569428998
babsi202
0
Netherlands
The Netherlands
NL
00
52.25
5.75
2024-12-24 10:22:13
2024-12-24 04:00:00
404992844
TomBauser
0
Frankfurt
Germany
DE
Hesse
05
50.11552
8.68417
2024-12-24 10:22:13
2024-12-24 04:00:00
903288533259083778
DemProud
0
United States
-14.60485
-57.65625
2024-12-24 10:22:12
2024-12-24 04:00:00
520658313
CharlesHAllison
0
New York City
United States
US
New York
NY
40.71427
-74.00597
2024-12-24 10:22:12
2024-12-24 04:00:00
2786988640
EricLebedel
0
Paris
France
FR
Île-de-France
11
48.85341
2.3488
2024-12-24 10:22:11
2024-12-24 04:00:00
Narratives Assets
Now that we have the conversations and posts, we classify them into narratives and discourse types.
🤖
The platform performs many data treatments with the help of AI agents. I’ve covered those extensively elsewhere, so here are a few resources to learn more:
Let’s first have a look at the lineage between those narratives assets:
We start by classifying the conversations as to whether or not they discuss climate-related topics. Let’s see some attributes:
Let’s also have a look at some of its metrics:
Let’s get a sample of data for those classifications:
CONVERSATION_ID
CLASSIFICATION
PARTITION_TIME
1867074284352393331
True
2024-12-12 10:00:00
1867050503504318545
False
2024-12-12 10:00:00
1867073426214519260
True
2024-12-12 10:00:00
1867075620628300041
True
2024-12-12 10:00:00
1867073189039276360
True
2024-12-12 10:00:00
Then we summarize the events being discussed in those conversations.
And we have the following metrics for this asset:
And a sample of data:
CONVERSATION_ID
EVENT_SUMMARY
PARTITION_TIME
1867041079163306152
On December 11, 2024, the Supreme Court issued a p
2024-12-12 07:00:00
1867022812025602097
A recent analysis highlights the significant benef
2024-12-12 07:00:00
1865150335284662433
In response to escalating geopolitical tensions, p
2024-12-07 04:00:00
1865181471918420443
In response to escalating threats from Russia and
2024-12-07 04:00:00
1865124092875067832
In response to escalating geopolitical tensions, p
2024-12-07 01:00:00
Finally, we associate each post to a discourse type and extract a narrative from it. Some attributes:
And some metrics:
Let’s get a sample of that data:
POST_ID
DISCOURSE_TYPE
NARRATIVE
PARTITION_TIME
1867041083688906784
Critical
This post discusses a significant legal ruling that reflects the ongoing struggle between environmental regulation and economic interests, particularly related to coal usage. The Supreme Court's decision to reject the Kentucky electric utility's request to block the EPA's efforts illustrates the critical dimension of climate change discourse, where economic and political power structures are challenged in favor of public health and environmental protection. It emphasizes the need for robust regulatory frameworks to manage hazardous materials like coal ash, which are direct byproducts of fossil fuel consumption, and highlights the societal implications of maintaining such harmful practices.
2024-12-12 07:00:00
1867022813896323229
Integrative
The post highlights the health and economic benefits of adopting heat pumps, framing climate change as an issue that intertwines environmental and social dimensions. By presenting data on reduced premature deaths, hospital visits, and asthma attacks, the discourse suggests that improving energy efficiency and transitioning to cleaner technologies not only addresses climate change but also significantly enhances public health and economic wellbeing. This integrative perspective encourages a holistic view of climate solutions, emphasizing the need to change societal norms and practices towards sustainable energy use.
2024-12-12 07:00:00
1854325713236611518
Critical
The post questions whether China adheres to the climate agenda, which reflects a critical discourse. It implies skepticism about China's commitment to international climate agreements, possibly due to perceived economic and political interests that may not align with aggressive climate action. This aligns with the broader narrative that addressing climate change requires challenging existing economic systems and power structures, as these can lead to uneven and unsustainable patterns of development and energy use. The post's context, following the election of Donald Trump, who has expressed intentions to roll back climate regulations, further underscores the tension between political actions and global climate commitments.
2024-11-07 04:00:00
1854377519069151413
Critical
This post highlights the social and economic implications of climate change, criticizing the lack of action to mitigate its effects despite the clear evidence of its future costs. The mention of 'our kids & grandkids' underscores the intergenerational impact of climate inaction. The tone suggests frustration with the current economic and political systems that prioritize short-term gains over long-term sustainability, which aligns with the Critical discourse. This reflects a challenge to power structures that maintain high fossil fuel consumption and neglect the urgency of climate policies.
2024-11-07 04:00:00
1854336568451797187
Critical
The post reflects a critical discourse on climate change in the context of Donald Trump's election as President. It implies dissatisfaction and frustration with the electoral outcome, suggesting that the country's leadership is not conducive to addressing climate change. This aligns with the critical discourse type, where climate change is seen as a social problem exacerbated by political structures and leadership decisions that prioritize economic gains from fossil fuels over sustainable development. The post echoes concerns that Trump's policies, which include dismantling climate regulations and promoting fossil fuel production, are at odds with the urgent need for climate action.
2024-11-07 04:00:00
Analytics Assets
Finally, the platform produces a dimensional representation of those assets for reporting purposes. I won’t go into details here, but here’s an overview of the dbt project that generates those assets:
📈 Conclusion
No sexy graphics, no groundbreaking insights. But isn’t that 90% of the work for data product builders? Designing the product, putting the pieces together, setting guardrails, and ensuring data quality.
There’s so much left to improve, but we at least now have a foundation to build upon. And most importantly, some data to explore new corners of our society. Because that’s the ultimate goal of RepublicOfData.io - to explore the dark corners of our society with data. And to get better at the craft of data product building along the way.