Blog

As Earth Day approaches, organizations shed light on any and all sustainability work they can claim credit for. This had me thinking about how sustainability is impacted by the technology center and how our ever-increasing digital usage contributes to the environment. In our digitally-driven world, computing resources have become essential for both personal and professional tasks. However, the convenience of computing comes with a hidden cost: environmental impact. From data centers to AI algorithms, our reliance on computing infrastructure contributes to carbon emissions and energy consumption, raising the question: Is sustainable computing an oxymoron?<h3>The Hidden Costs of Computing Resources</h3>Data centers, the backbone of modern computing, consume vast amounts of energy. According to the International Energy Agency (IEA), data centers consumed around 200 terawatt-hours (TWh) of electricity in 2020, accounting for about 4% of global electricity demand. This consumption is projected to increase as digitalization expands.Email, seemingly innocuous, also has an environmental footprint. Storing emails on servers requires energy, and as our digital communication grows, so does the energy consumption. A study by the University of Michigan found that storing one gigabyte (GB) of data in the cloud can produce up to 4.1 kg of carbon dioxide equivalent (CO2e) per year.Artificial intelligence, particularly generative AI, poses additional challenges. Training large AI models demands significant computational power, often fueled by energy-intensive CPUs and GPUs. OpenAI's GPT-3 model, for instance, reportedly emits over 626,000 kg of CO2e during training, equivalent to nearly five times the lifetime emissions of the average car.<h3>The Challenge of Collective Action</h3>Like most challenges with tackling sustainability issues, the problem is built up from 1000s miniscule decisions that add up to an enormous impact. Given the small impact on individual decisions it is hard to weigh the importance of sustainability when the impact at the individual level is so low. It takes extreme organization and unified action across multiple parties to put a dent in our environmental impact. While individual organizations may recognize the importance of reducing their environmental footprint, navigating the distributed nature of sustainability poses significant obstacles to achieving meaningful change. As there is limited low-hanging-fruit or areas where one change could make a drastic impact, organizations must create unified action across their domain.&nbsp;<h3>The Path to Sustainable Computing</h3>While the environmental impact of computing is undeniable, there's hope for a sustainable future. Adopting energy-efficient practices, such as optimizing data center cooling systems and transitioning to renewable energy sources, can mitigate the carbon footprint of computing infrastructure. Innovations in AI, such as more efficient algorithms and hardware acceleration, offer promising avenues for reducing energy consumption. Make sustainability a priority: To ensure sustainable computing practices, it's imperative for company leaders to prioritize environmental concerns at the core of their business strategies. This entails integrating sustainability into the company's vision, mission, and decision-making processes, with clear goals and objectives set to reduce environmental impact. Implementing robust policies and governance structures is essential to enforce sustainability practices across all levels of the organization, fostering accountability and ensuring alignment with environmental objectives.Empower conscious consumption:&nbsp;Encouraging individual contributors to make environmentally conscious choices is key to driving sustainable computing practices. Providing transparent information about the environmental impact of computing products and services enables consumers to make informed decisions based on sustainability criteria. Establishing eco-labeling and certification schemes can further incentivize companies to adopt sustainable practices while enabling consumers to easily identify eco-friendly options. Build in optimization for continuous improvement:&nbsp;We know technologies are constantly changing - so we should embed sustainability into our practices so as we change technology stacks, we do not lose our sustainability efficiencies. Continuous optimization is essential for improving the environmental performance of computing operations over time. Conducting comprehensive lifecycle assessments allows organizations to evaluate the environmental impact of their products and services and identify areas for improvement. By implementing real-time monitoring systems, organizations can track energy consumption, carbon emissions, and environmental performance, enabling them to proactively identify opportunities for optimization.&nbsp; Go beyond renewable resources: While renewable energy sources are crucial for reducing carbon emissions, it's essential to recognize that energy itself remains a finite resource. Therefore, organizations must prioritize energy efficiency measures and optimization strategies to minimize energy consumption and maximize resource utilization, even when relying on renewable energy sources. Embracing principles of resource conservation and circular economy practices can further minimize waste generation and maximize the reuse, recycling, and make IT systems even more resilient. By fostering innovation and collaboration with industry partners and research institutions, organizations can drive advancements in sustainable computing and develop technologies that reduce dependency on finite resources while ensuring a greener future for computing operations. <h3>The Importance of Conscious Design</h3>As we advance technologically, it's crucial to prioritize sustainability in computing infrastructure. Designing with environmental consciousness can minimize our impact on the planet. This includes optimizing software for energy efficiency, adopting circular economy principles for hardware, and investing in renewable energy.Sustainable computing is not an oxymoron; it's a necessity. By acknowledging the environmental impact of computing resources and implementing sustainable practices, we can mitigate our carbon footprint and build an eco-friendlier digital ecosystem. However, it requires collective effort—from individuals to corporations—to prioritize sustainability in technology development.By Ellie Najewicz

Is Sustainable Computing an Oxymoron?

In the buzz of AI, vector databases have emerged as a focal point of attention, especially within the context of generative AI (Gen AI). While the hype surrounding vector databases may seem daunting, it's essential to demystify their significance and understand their role in powering next-generation AI applications. In this blog, we'll explore the origins of vector databases, their relevance in Gen AI, and best practices for leveraging them effectively.<h3>Understanding Vector Databases</h3>Contrary to popular belief, vector databases are not a novel concept but rather an evolution of existing databases and AI technologies. At their core, vector databases specialize in storing and querying high-dimensional vectors, which encapsulate numerical representations of data points in multi-dimensional space. Basically, a vector is a numeric representation of unstructured data that can be easily searched and identified as similar. A vector database has collections that essentially holds two components (1) the unstructured data (text, audio, image, etc.) and (2) it's vectorized value.<figure id="figure-061p1yxaallu9xabi3"><img id="061p1yxaallu9xabi3" src="https://cdn.durable.co/blocks/fVZd1f0FJC3QMmZhtTbs9eRG3Z72t5xOLHZsKAIkKyQOIQJK9kxjZfRPZMViTgIL.png" alt="Vector databases" title="Vector databases" class="object-cover bg-gray-100 " style="aspect-ratio: 3/2; object-position: undefined undefined;" data-corners="default" data-aspect="3:2" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/fVZd1f0FJC3QMmZhtTbs9eRG3Z72t5xOLHZsKAIkKyQOIQJK9kxjZfRPZMViTgIL.png" data-mce-style="aspect-ratio: 3/2; object-position: undefined undefined;"></figure>These vectors serve as the building blocks for a wide range of AI applications, including natural language processing, image recognition, recommendation systems, and more. Models essentially search a vector database to identify data that is similar to the prompt to understand what is being requested and how to generate material based on historic responses.<h3>The Role of Vector Databases in Gen AI</h3>With Gen AI, where artificial intelligence intersects with human-centric experiences, vector databases play a pivotal role in enabling transformative innovations. By harnessing the power of vector representations, AI systems can understand, interpret, and contextualize complex data sources quickly, thereby facilitating more intuitive and personalized interactions with users. For example, in natural language processing, vector databases enable semantic search and similarity matching, allowing AI systems to comprehend the underlying meaning and context of textual data. Similarly, in image recognition, vector databases facilitate feature extraction and comparison, empowering AI models to identify patterns and objects with remarkable accuracy.What do vector databases really offer? &nbsp;What is actually stored in a vector database is relatively simple, however the value a vector database provides is improved searchability specifically for searching vector values. The indexing logic is what separates vector databases from traditional databases - that usually rely on b-tree logic for indexing which won't support vector search. So, while you could store vector embeddings in a non-vector database, you would not be able to search them effectively.<h3>Best Practices for Leveraging Vector Databases</h3>While vector databases hold immense potential, it's crucial to adhere to best practices to maximize their effectiveness and mitigate potential challenges:Data Modeling and Vectorization: Invest time and effort in designing robust data models and vectorization strategies tailored to the specific requirements of your AI applications. Many principles already established for NoSQL can be applied as searches against a vector database need to be kept as simple as possible (i.e. no joins please). Consider factors such as feature selection, dimensionality reduction, and normalization to optimize vector representations for efficient storage and retrieval.Scalability and Performance: Choose vector database solutions that offer scalability and performance optimizations to accommodate growing datasets and demanding workloads. Evaluate factors such as indexing techniques, query optimization algorithms, and distributed computing capabilities to ensure seamless scalability and efficient query processing.Data Quality and Governance: Prioritize data quality and governance practices to ensure the integrity, consistency, and reliability of vector data stored in the database. However, this must be implemented before the data is vectorized. The data held in a vector database should already be clean and governed. If you are trying to implement governance on vectorized values, you are already too late. Implement data validation, cleansing, and monitoring mechanisms to detect and rectify errors or anomalies in vectorized data, thereby enhancing the trustworthiness of AI-driven insights.Interoperability and Integration: Foster interoperability and integration between vector databases and other components of your AI ecosystem, such as machine learning frameworks, data pipelines, and visualization tools. Adopt standardized protocols, APIs, and data formats to facilitate seamless data exchange and interoperability, enabling cohesive and synergistic AI workflows.In conclusion, vector databases are instrumental in powering next-generation AI applications, offering unparalleled capabilities for storing, querying, and analyzing high-dimensional vector data. Popular vector databases include Milvus, Chroma, or Pinecone. Also, existing databases that support other data structures have added vector support such as PostgreSQL and MongoDB. By adhering to best practices and leveraging vector databases effectively, organizations can unlock the full potential of Gen AI, driving innovation, and delivering transformative experiences for users. Embrace the possibilities of vector databases as a cornerstone of your AI strategy and embark on a journey towards AI-driven excellence and impact.&nbsp;By Ellie Najewicz

Vector Databases: Navigating the Hype

In the ever-evolving landscape of technology, the role of a technical architect is paramount in shaping robust and scalable solutions. While innovation drives progress, it's crucial to recognize the inherent value of reuse and building upon existing foundations. In this blog, we'll delve into the importance of reuse and the pitfalls of constantly reinventing the wheel.<h3>The Case for Reuse</h3>Reuse is not merely a cost-saving measure; it's a strategic imperative that fosters efficiency, reliability, and scalability in software development. Here's why technical architects should prioritize reuse:Accelerated Development: Leveraging existing components, frameworks, and libraries accelerates development cycles, allowing teams to focus on solving unique business challenges rather than reinventing basic functionalities.Consistency and Reliability: Reuse promotes consistency and standardization across projects, reducing the likelihood of errors and inconsistencies. By building upon proven solutions, technical architects can enhance reliability and minimize risks.Scalability and Maintenance: Reusable components are inherently scalable and adaptable to evolving requirements. They facilitate easier maintenance and updates, ensuring long-term viability and sustainability of software systems.Knowledge Transfer and Collaboration: Reuse encourages knowledge sharing and collaboration among team members. By utilizing shared components and patterns, technical architects can foster a culture of learning and innovation within their organizations. Continuously enhancing existing solutions is a great way to foster work across organizational boundaries - as opposed to creating several competing solutions.&nbsp; <h3>The Pitfalls of Building Something New</h3>While innovation is essential, constantly building something new can lead to several shortcomings such as increased complexity, duplication of work, and a fragmented organization. Building from scratch often results in unnecessarily complex solutions, leading to higher maintenance costs, longer development cycles, and increased technical debt.&nbsp;Additionally, it is common issue to find repeatedly reinventing the wheel, which leads to duplication of effort and resources, squandering valuable time and resources that could be allocated to more strategic initiatives.Having a always build new mindset can lead to a fragmented ecosystem across an organization. A proliferation of custom-built solutions fragments the technology ecosystem, making it challenging to maintain consistency, interoperability, and compatibility across projects. And within these new developments, they carry inherent risks, including unforeseen technical challenges, integration issues, and compatibility constraints. By relying on proven solutions, technical architects can mitigate these risks and ensure project success.<h3>Embracing Reuse as a Guiding Principle</h3>We have all heard of the importance of challenging the status quo, and pushing back when someone says "that's the way we have always done it" or "why fix what isn't broken". While there are situations where building something new is necessary, it should be the exception rather than the norm. Technical architects should advocate for a culture of reuse within their organizations, emphasizing the following principles:1. Evaluate Existing Solutions: Before embarking on new projects, conduct a thorough evaluation of existing solutions and identify opportunities for reuse. Leverage industry best practices, open-source libraries, and internal repositories to identify reusable components. Before anything new is created, there must be clear evidence and documentation that existing solutions did not exist or could not cover the use case. 2. Standardization and Documentation: Establish standards, patterns, and best practices for reuse within the organization. Document reusable components, APIs, and design patterns to facilitate knowledge sharing and collaboration among teams. Standards and patterns should be easily searchable within an organization, so it is easy to foster reuse. 3. Continuous Improvement: Foster a culture of continuous improvement by soliciting feedback, conducting retrospectives, and refining reusable components based on lessons learned. Encourage developers to contribute enhancements and optimizations to shared libraries and frameworks.4. Strategic Innovation: Reserve resources for strategic innovation and differentiation, focusing on areas where custom-built solutions offer a competitive advantage. Strike a balance between innovation and reuse, ensuring that new initiatives align with business objectives and organizational priorities.In conclusion, reuse is a guiding principle that empowers technical architects to build scalable, reliable, and maintainable software solutions. By prioritizing reuse and building upon existing foundations, technical architects can accelerate development cycles, enhance reliability, and foster a culture of collaboration and innovation within their organizations.

The Pitfalls of Always Building Something New

a black background with red and green letters

In the vast expanse of digital information, there exists a shadowy term called dark data – a trove of unstructured or unutilized data lurking within organizational repositories. Despite its ominous name, dark data harbors untapped insights and potential, making it a valuable yet often overlooked resource for organizations. <h3>What is Dark Data?</h3>Dark data encompasses information assets that organizations collect, process, and store during regular business activities but fail to leverage for insights or decision-making. This data often remains dormant, residing in siloed systems, forgotten databases, or unstructured formats such as emails, documents, and multimedia files. Dark data takes many forms, and can include incomplete data, unlabeled data, improperly formatted data, expired data, biased data, or even data that has yet to be discovered. Unlike structured data, which can be readily analyzed and utilized, dark data languishes in obscurity, obscuring valuable insights and opportunities.<h3>Dangers of Dark Data</h3>Understanding and harnessing dark data is imperative for organizations seeking to remain competitive and innovative in today's data-driven landscape. There are multiple implications for the ongoing maintenance, storage, and ingestion of dark data. To fully understand these concepts, let us unpack some of the major impacts of dark data:1. Optimization of Resources: Large amounts of dark data lead to poor data usage and management of storage resources. Dark data does not provide business value, but still maintains costs to store, secure, and manage. Dark data stands in the way of storage consolidation and system optimization for any organization. 2. Carbon Impact of Data: The carbon impact of dark data is an emerging concern, with organizations projected to emit significant CO2 due to its presence. Sustainable data management solutions can mitigate this impact, aligning with organizations' environmental responsibilities. In 2020, study conducted by Vertitas Technology had projected that in that year 6.4M tons of CO2 will be pumped into the environment due to the presence of dark data.3. Accuracy of Data Analysis: The accuracy of data analysis is contingent upon the accuracy of the data itself. Organizations must discern critical data for their business requirements to ensure trustworthy insights. Moreover, considerations such as data governance and privacy models influence the acceptance and trustworthiness of transformed data.4. Increased Privacy and Security Risks: Unmanaged dark data poses a significant risk in terms of compliance, security, and privacy. Regulatory frameworks such as GDPR and CCPA mandate organizations to ensure the proper management and protection of all data. By nature, dark data is difficult to protect as it usually off-the-radar and organizations cannot protect data assets that they cannot detect or interpret.&nbsp; <h3>Hidden Opportunities</h3>All organizations have an opportunity to reduce the amount of dark data impacting their business. Let's unpack what steps we can take to correct or eliminate dark data:- Assessment and Inventory: Conduct a thorough assessment to identify dark data and categorize its sources. Develop an inventory to track the types and locations of dark data, and plans to resolve. Based on the risk the data imposes by staying dark, tackle the most critical cases of dark data for best results. - Data Governance and Transformation: Implement robust data governance frameworks to ensure transparent and ethical data transformation processes. Especially for dark data that is incomplete or unlabeled, through governance this data can be identified and resolved. Governance is also the best measure to make sure new data in an organization does not become dark. - Prioritize through Sustainability: Foster collaboration between data and sustainability teams to integrate environmental considerations into data management practices. Invest in innovative technologies and talent to maximize the value of transformed dark data while minimizing risks.Overall, by focusing on dark data and embracing sustainability, organizations can unlock valuable insights, mitigate risks, and contribute to a more environmentally responsible future. Through conscientious management and transformation, dark data can illuminate pathways to growth and sustainability.

Shedding Light on Dark Data

&nbsp;In today's interconnected world, "privacy" is frequently seen as a pillar of "security", yet they represent distinct concepts crucial for safeguarding sensitive information. Since privacy impacts many professions, it's imperative to unravel the disparities between these two programs and understand their unique significance, especially within the context of business operations.<h3>Navigating Nuances</h3>At its core, any textbook and common security practices tell you security encompasses a triad of principles: confidentiality, availability, and integrity. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Confidentiality ensures that data is accessible only to authorized users, shielding it from unauthorized access or disclosure. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Availability guarantees that data is accessible to authorized users whenever needed, without compromise. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Integrity ensures that data remains accurate, complete, and unaltered throughout its lifecycle. Security measures such as encryption, access controls, and firewalls serve as protections against a myriad of threats aiming to compromise either confidentiality, availability, or integrity.Conversely, privacy primarily revolves around confidentiality, emphasizing the individual's right to control the collection, use, and dissemination of their personal data. It's about fostering trust and transparency by respecting individuals' autonomy over their information. Privacy regulations such as the GDPR (General Data Protection Regulation) and CCPA (California Consumer Privacy Act) delineate stringent guidelines governing the handling of personal data, mandating consent mechanisms, data minimization practices, and robust data protection measures. While the two are separate practices they are interdependent. Privacy depends on personal data being secure - as a lack of security compromises a business’s responsibility to protect private information. Also, security depends on the privacy of credentials that are needed to implement access control across a system.&nbsp; <h3>Interdependent Approach</h3>Businesses must recognize that while security is indispensable, privacy augments its essence by prioritizing individuals' rights and expectations regarding their personal information. Consequently, the approach to privacy necessitates a more nuanced and empathetic perspective, rooted in ethical considerations and respect for individual autonomy. Therefore, organizations should have two separate offices to enforce privacy and security. Keeping privacy as part of security, will limit the reach and priority of privacy requirements. To effectively navigate the complexities of privacy, businesses should adopt a proactive stance, integrating privacy-by-design principles into their operations. This entails embedding privacy considerations at the outset of product development and service delivery, rather than treating them as an afterthought. Conducting comprehensive privacy impact assessments and implementing privacy-enhancing technologies fortify the organization's resilience against privacy risks, fostering a culture of accountability and trust. By championing privacy as a fundamental human right, businesses not only mitigate legal and reputational risks but also cultivate enduring relationships based on mutual respect and trust.In conclusion, while security and privacy are intertwined, they cover distinct areas crucial for safeguarding sensitive information in today's organizations. By embracing privacy as a core value and integrating it into their operations, businesses can forge a path towards sustainable growth and resilience in an ever-evolving landscape of privacy concerns and regulatory scrutiny.

The Difference Between Security and Privacy

Storing null values is an important feature of any database. Knowing if data is stored is an essential operation to quickly sort and retrieve data. Providing your application function to pass nothing or no information is key to many workflows where partial or incomplete transactions should be persisted. Let's take a look into how to best leverage null values to optimize or database. <h3>The Silent Players </h3>Why should we care about storing nothing?&nbsp; Knowing that nothing has been stored for a particular column or field is just as important as understanding the value if something was stored. Thinking about how your database will handle null values should be defined when you first build your data model as well as when you are designing your queries to interact with the data. Additionally, nulls are key to understanding if data can be used for analytics or AI development.<h3>All Upside </h3>There are so many good things from using null values properly. This way databases can efficiently hold data that is empty or partial information to accurately reflect a business process. Nulls can provide the following:&nbsp;- Query performance improvements: Null values provide database professionals with the flexibility to index columns efficiently. This allows for faster search operations, as nulls are indexed separately, contributing to improved query performance.- Better data quality: By utilizing null values, database professionals can avoid the introduction of misleading information. Nulls serve as clear indicators of missing data, preventing misinterpretation, and ensuring that decisions based on the database are made with a comprehensive understanding of the available information.- Enforce referential integrity: Null values provide a controlled way to delete or update records without violating referential integrity constraints. Instead of forcing cascading updates or deletions, nulls permit a level of flexibility, allowing for the maintenance of relationships without compromising the integrity of the database. <h3>Not all Nulls are Equal</h3>The biggest issue that can arise with null usage is not setting them correctly. Sometimes team can insert 'NA', '&nbsp;&nbsp; ', 'BLANK', or other strings to indicate an empty value. This will have serious consequences on data quality and database performance. As one of these examples is still a string - so it takes up storage and must be read explicitly by queries when searched. The largest mistake people make with numeric numbers is treating a value of zero as NULL - because these values are not the same. Null numeric values of zero can skew computations and lead to misinterpretation in reporting. For example, a contract valued as NULL is a contract where the price has yet to be determined or is in negotiation. Whereas a contract with a value of zero means a contract is being completed free of charge or not for a monetary value. If both of these scenarios, were stored with a value of zero - you could see how that would lead to misinterpretation. In conclusion, leveraging nulls strategically empowers technical teams to build robust and efficient database systems that not only store and retrieve data effectively but also maintain the integrity of relationships between tables. By embracing null values as valuable contributors to the database ecosystem, we can harness them to create more resilient and adaptable systems.By Ellie Najewicz