Categories
Blockchain Financial Services

A practical guide to Hyperledger Fabric blockchain security

Blockchain is definitely still a buzzword in the IT world. Uses beyond cryptocurrencies continue to intrigue the business world. But how can you be sure of Hyperledger Fabric security?

Stories of hacked crypto stock exchanges or their users are a never-ending story. This makes me suspect that many don’t know what they’re really doing while setting up blockchain infrastructure, and don’t fully grasp the security implications. This article is to help people interested in blockchain technology and who want to understand its security aspects and take appropriate security measures.

Specifically, this article covers important security aspects of the Linux Foundation’s Blockchain Hyperledger Fabric, which is the main blockchain platform, as well as Apache Kafka, and Apache ZooKeeper. Such permissioned networks as Hyperledger require these auxiliary tools for messaging and service discovery.

 

A practical guide to blockchain Hyperledger Fabric security

Table of contents:
  1. Proper networking
  2. Blockchain security – how to do it properly
  3. Process/application isolation and resource accounting
  4. Docker
  5. Log collection and analysis
  6. Proper backup data
  7. Hyperleder Fabric security
  8. Kafka and ZooKeeper functional security
  9. Conclusion

 

 

Proper Networking

Developers often make the mistake of trying to implement blockchain for business Hyperledger Fabric security themselves. They focus on its functional security while leaving basics such as network or firewalling open to potential attacks. There is, however, a reason. Configuring a blockchain stack is far more complicated than configuring a MySQL server, to name one example.

Typical Hyperledger/ Kafka/ ZooKeeper installation uses over 10 different ports on different types of hosts:

  • 7050 for Orderer (central service)
  • 2181, 2888, 3888 for all ZooKeeper instances
  • 9092 for all Kafka instances
  • 9000 for Kafka Manager (central service)
  • 7054 for all Fabric Certificate Authority (CA) instances
  • 7051, 7053, 8053 for Peers
  • 2377, 7946, 4789 for Docker swarm

Most of these components are for internal use, so you don’t need to expose them outside your Hyperledger server node. Peers, on the other hand, link together over a P2P network, therefore, they need to see each other. Of course, you can change default port numbers. And clients connect directly to Peers, Certificate Authorities and Orderer.

Since each component can deploy inside NAT or even inside an internal container, mostly Docker or LXC, each component can also use custom external ports to connect to discover and connect to each other. For example, the Chinese Alibaba Cloud implements this by using ports 32050, 32060, and 32070 for Orderer service.

Even when exposing all services on default ports, there are a lot of network connections between components. There are also firewall rules to write and manage. This is precisely why many people surrender proper network security of their Hyperledger installations and pass all traffic between all “trusted” hosts. Obviously, this approach is only as secure as these hosts are trustworthy.

 

Blockchain security – how to do it properly

There are several approaches to make blockchain systems secure. However, all of them are based on generating lists of IP ranges dynamically. You can do this either by simple shell script iterations and string concatenations or by compiling any form of iptables/ebtables profiles. Honestly, any other method specific to a chosen firewall solution that allows merging IP rules with port rules will work. Passing traffic through several chains, for instance.

A suggested approach is to make a three-layer solution where:

  • the first layer is based on AWS VPC or Direct Connect assuming that the whole solution runs on AWS. It just narrows access to defined port numbers
  • the second layer is a host-based firewall iptables, possibly wrapped by your chosen management solution. This restricts access to given IP ranges, constructed dynamically
  • the third layer is also host-based, imposing granular rules per IP range and service

Why divide the same set of rules into two separate levels? This method has two advantages:

  • splitting rules management into separate streams: trusted hosts management and Hyperledger services management
  • additional protection in case of human mistakes. Setting too wide an access at one level doesn’t open your Hyperledger installation to attacks

If you use Docker, a good idea is to integrate the second level rules with Docker networking rules. Do this by replacing default PREROUTING rules by similar rules restricted to given IP ranges. More on that in my GitHub repository.

And, of course, services used internally within the same host should be the only place that exposes it. Think ZooKeeper nodes. Listening only on 127.0.0.1 is also a good idea. If you use Docker, use the Docker local network.

 

 

 

Process/application isolation and resource accounting

Hyperledger application stack is quite complicated:

  • Hyperledger Fabric’s main component is in Go
  • Apache Kafka is in Scala (runs on JVM)
  • Apache ZooKeeper is in Java
  • Apache CouchDB (optional Peer state database) is in Erlang

Each component has a different configuration style, low-level software dependencies such as a specific version of the Go stack, and details related to resource accounting. Therefore running everything on a single host is not a good idea. I recommend two approaches:

  • a static approach, based on LXC containers (LXC is the same paravirtualization technology that Docker uses. However, LXC containers are persistent and act more like virtual machines, except without resource reservation, preferably using Proxmox
  • a dynamic approach, based on Kubernetes, but I don’t recommend bare Docker for such complicated stack, except for the development phase

What’s the difference between these approaches? The first one is better for running small-size, but long-running setups for internal purposes like integrating blockchain-based applications — debugging Hyperledger on LXC in much easier than on Docker. And of course, it’s much easier to implement the proper firewalling scheme, as I mentioned above.

What LXC lacks is easy scalability and that’s why you should run production Hyperledger using Kubernetes. Both Proxmox and Kubernetes have built-in resource accounting:

hyperledger fabric security
Source: Proxmox docs

At the time of writing this article, Amazon just announced Go language support in AWS Elastic Beanstalk. In the near future, this may be a nice alternative to LXC for setting up small setups for staging/integration/pre-production purposes. I will be watching this development.

 

hyperledger fabric security
Source: Kubernetes docs
 

Docker

You can also set up the whole Hyperledger stack using Docker. This is arguably a preferred solution for developing Hyperledger-based applications. There are two basic recommendations for such a deployment:

  • run processes within containers as a non-root user (this part requires preparing such configuration in Dockerfile), and start containers themselves as non-root — this is to prevent exploitation in case you discover new bugs similar to CVE-2016-9962 or CVE-2019-5736
  • control who has access to “docker” system group (and can manage containers on the host)

Of course, similar rules apply to all Hyperledger environments, no matter which platform you choose: control system-level access (eg. to prevent unauthorized copying of Hyperledger internal files, and perform at least some basic security hardening.

 

Log collection and analysis

Hyperledger Fabric provides several options to fine-tune logging so you can easily configure it to coexist with your chosen log collection and analysis solution. Think Splunk, ELK or EFK, or even logcheck/logwatch.

But which log analysis solution to choose? The one, that you or your team are most familiar with. If none, then ELK (Elasticsearch + Logstash + Kibana) should be the safe choice — it’s both popular and has pretty decent functionality.

What you should bear in mind at this stage, is the proper log file handling, especially if you run Hyperledger stack on bare Docker. A common mistake here is redirecting logs to/dev/null, which causes these logs to be lost. Instead, dump logs to files and import to ELK using Filebeat or parse them directly with tools such as logcheck.

 

Proper data backup

There are two different approaches to build a proper data backup/restore solution for Hyperledger Fabric:

  • just backup all contents of /var/Hyperledger/production directory — there is lots of data there in a real production installation, so this approach might seem superfluous. However, it’s not. Data storage is relatively cheap compared to the many hours of work from someone who knows Hyperledger Fabric security fundamentals
  • handle Peer transient storage separately from ledgers, chains, or private data — this can raise the backup efficiency and lower the costs. Be aware, though that it requires deep (and current!) knowledge of Hyperledger Fabric internals to implement it properly. – or else, it could introduce a risk that such a backup would be incomplete and unusable, leaving us effectively without a backup

If you already have several people with Hyperledger Fabric security knowledge, then maintaining such a complicated (but more efficient!) backup should not be a problem. However if not, then my recommendation is to put more money into data storage to avoid the risk of losing data.

If you have enough experienced people who understand Hyperledger Fabric security architecture and its implications regarding data backup, restore and encryption, you can implement data encryption with internal the Hyperledger Fabric encryption library — so only peers with decryption keys can use internal files.

File encryption increases overall security, though it degrades backup compression and performance — and can break GDPR compliance. Fully automatic data recovery is more complicated. There is no mechanism for automatic reapplying of data where users have invoked their right to be forgotten. Data encryption on this level makes it impossible to implement one — which is still ok for automated backups but prevents fully automated restore processes.

All the above security aspects are “just” low-level technical aspects. They exist more or less for any hosted IT environment. Now, let’s discuss some more functional aspects of Hyperledger Fabric itself.

 

Hyperledger Fabric Security

Connecting TLS encryption and proper certificate handling are the most important aspects of functional security. Blockchain data is secure by design, however, the functional part of this security relies on proper configuration of Hyperledger certificate authority (CA) with proper key management.

 

Hyperledger fabric security
Source: Hyperledger Fabric docs

Having the CA part configured, it’s time to set up Attribute-Based Access Control, which allows smart contracts to operate on specific client attributes. This, along with enabling TLS client authentication on peer nodes sets the overall trust level in the whole network reasonably high.

Of course, apart from the network level, there’s still the host level, on which malicious actors can steal data — at least unless it’s under encryption. So you should also consider the best method to encrypt it. You can use either Hyperledger Fabric native encryption or on the filesystem level such as LUKS or on a cloud provider level such as the  AWS Key Management Service. How to do it properly? It depends on your whole architecture, and, in particular, which layers you want to have fully automated, and which should require manual intervention in case of failure.

 

Kafka and ZooKeeper functional security

Securing Hyperledger itself doesn’t make much sense when underlying components (connection encryption, authentication, and authorization) are not secure. Also, don’t forget to properly secure access to the Kafka Manager panel — exposing it directly to the Internet is obviously a bad idea. Instead, you should put it behind some proxy, such as Nginx or HAproxy, that will also handle SSL termination.

Looking from a technical point of view, ZooKeeper is a simple pair of two TCP servers with some queues and distributed key-value store and a quorum algorithm, which is the heart of service discovery for all services relying on ZooKeeper. More on that here. This part is however more complicated since ZooKeeper functionality is much more than just a message broker.

As in Kafka, ZooKeeper needs a proper configuration of connection encryption (SSL — including keystore configuration on JVM level), authentication (SASL) and authorization (ACL).

ZooKeeper is harder to configure and maintain properly, since stability problems of particular instance chosen as Quorum Leader, lead to another election. The election process is fully automatic, however until the new Leader is chosen, ZooKeeper suspends its service discovery functionality.

Of course, a single failure such as a manual service start is absolutely not a problem. However, if many instances have random problems, or if your upgrade procedure instructs restarting all upgraded services at the same time, which is very common with most Puppet manifests for ZooKeeper, found on the Internet, then it can affect the stability of your whole network of services, not only the ZooKeeper quorum itself.

 

Conclusion

As you can see, ensuring Hyperledger Fabric security is not an easy task. In fact, it’s one of the hardest tasks in my almost four-year career at Espeo Software and over 20 years in IT overall.

There are lots of complex software running on different application stacks that use lots of data. All components are using service discovery instead of static configuration. Any infrastructure as code tool could easily manage it. Instead of this, you have a living, breathing, fragile service network. You also need a trained team that really understands the impact of their actions.

Having such people on board is something that distinguishes companies that think seriously about their blockchain business from ones that only want to sell the buzzword and run.

We are helping various businesses with powerful enterprise blockchain network based on Hyperledger Fabric. Click here to contact us and one of our experts will get in touch with you.

Related posts:

Categories
Entrepreneurship Software

Project management tips for ongoing projects

We’ve already covered project management tips for starting off, or how to best prepare for project kick-off. Once all the items have been checked off the list, you can get the ball rolling. However, the project manager’s work will continue for weeks — even months. Here’s what a good PM should keep an eye on.

Our second project management tips checklist should help you deal with the regular issues you’ll face as a PM. Alternatively, if you’re viewing this from the other side — this is what you should expect as a client. We’ve actually touched upon some of these issues in our previous checklist. For instance, we suggested that you establish reporting rules early on. Reporting remains an important matter in your weekly and monthly checklist. If reporting is handled well, both parties will be prepared for meetings, and will know what information is needed at a particular moment. Smooth communication and great contact with the client and transparency are very important. We’re fully agile, so transparency is a natural piece in the PM puzzle for us.

With 14 years on the market, we have gained valuable expertise in various areas. Read more about what we excel at.

Don’t forget about your team! They might have holiday plans, and these should be accounted for in advance. Remember to ask about team morale. Is there anything in the project that’s causing the team members to be unhappy and frustrated? What is it? It’s also good form to know.

If there’s anything you’d like to read about, let us know in the comments!

Project management tips for ongoing projects

See also:

Categories
Entrepreneurship Newsroom Software Technology

Espeo Software Proud to be Named a Top Development Partner in Poland by Clutch!

Here at Espeo Software, we know it can seem tricky, especially in these unprecedented times, to attract new customers while creating cutting edge solutions. We can help you do both! We’re a digital consultancy with a strong international presence. Our seasoned team builds and designs one of a kind custom web and mobile applications to keep our clients ahead of the game. We have the industry know-how and strong technical experience to help you deliver! 

Espeo Software Proud to be Named a Top Development Partner in Poland by Clutch!

In recognition of our success and prowess, we’ve been named a top Polish development partner by Clutch, a data-driven B2B platform. They use a unique reviews based formula to compare and contrast leaders across a number of sectors. Their findings help clients around the globe find a good fit for their latest service projects.

See also: Espeo Software Wins Clutch Award for Poland’s Finest Developers

We’d like to raise a special toast to our wonderful clients who helped us get this far. They took the time to engage with Clutch representatives to evaluate our impact. We were graded along the lines of timeliness, customer service, and overall project management skill. We’re thrilled to say we’ve been able to maintain a 4.7 out of five stars rating on Clutch, based on that customer feedback. Take a look at just one of our several reviews below:

Espeo Software Proud to be Named a Top Development Partner in Poland by Clutch!

“We’re honored to receive this recognition again. Client’s feedback is one of our most important metrics to ensure that what we do is outstanding and helpful.” – Head of Marketing, Espeo Software 

We’re dedicated to keeping our clients at the top of their game, and we’ll use this award to fuel our efforts even further!  Drop us a line if you’re interested in learning about how we can help your business soar today.

Categories
Blockchain Finance Financial Services

The Top 10 Best Crypto Blogs and Personalities to Follow for 2022

As blockchain technology percolates deeper into almost every sector of our economies — from finance to supply chain to real estate, there is more to this revolutionary technology than algorithms, regulation, and code. The more blockchain use cases transform our life financial markets, the more sources of information regarding this technology appear on the internet. In order to help you navigate the crypto industry, we have prepared a list of the top blockchain blogs out there.

If you are tired of the constant price speculation in sources, such as Reddit, Telegram groups, and the more mainstream crypto websites, here are some of the best sources for coverage of blockchain news and crypto markets authored by those with their fingers on the pulse of the decentralization movement.
 

The Top 10 Cryptocurrency Blogs and Crypto Influencers to Follow

Table of contents:

 

CoinDesk

CoinDesk is one of the most popular news platforms for such crypto currencies as Bitcoin, Ethereum, and altcoins today. In fact, other crypto websites cite it as a source of their information. However it is a very insightful content, so that Cryptocurrency newbies may encounter difficulties understanding some of the terminologies used in their blog posts.
It is run by CoinDesk Inc., which carries out one of the most popular events of the crypto space — the Consensus Summit, and daily publishes analytical content touching all blockchain uses cases. Generally, CoinDesk is one of the most popular and best crypto blog handling cryptocurrency news and anything affecting the blockchain industry.

Cointelegraph

Cointelegraph launched more than seven years ago and is entirely autonomous. The platform publishes a lot of insightful content regarding crypto markets, world blockchain events, collections of altcoins, and general information of mining activities. They cover four primary topics: cryptocurrency trading, blockchain, mining activities, and crypto and blockchain news.
Remarkably, they discuss digital currencies from all angles, comparing them to the standard for payment transactions, the value measure, and the medium of exchange. Cointelegraph looks for use cases of blockchain in the real-world and analyzes everything that happens in the crypto world. This includes a clear structure, a news feed, crypto rating, regulation, and information regarding upcoming coin offerings.

Vitalik Buterin’s Blog

It is hard to condense all of Vitalik Buterin’s contributions to the blockchain space into a single blog. He successfully managed to create the Ethereum blockchain, its native currency, and its native programming language. Because of the success of the Ethereum network, especially opening the way for the creation of decentralized applications, Vitalik is one of the crucial personalities in the evolution of blockchain.
In his crypto blog, Vitalik discusses everything from the history and challenges of the Ethereum blockchain to the development of theories on levels of fault-tolerant consensus and even how the blockchain industry should be regulated. If there is any crypto news website that you should consider reading as an investor looking for insightful content that will help you to gain a more technical understanding of blockchain technology, this should be among your priorities.

CCN

Launched in 2013 as Cryptonews.com, CCN provides not only news about cryptocurrencies and crypto markets, but also detailed analysis. The content is supported with clear graphs and figures, offering valuable information about trade volume, exchange rates, and changing market trends. Focused initially on Bitcoin, the blog has broadened its content offering to cover the current, expansive coin offerings in the crypto space.
The brand has established itself as a source of reliable cryptocurrency information. The crypto community likes their way of presenting information- structured, briefly, and mostly, not forgetting their provision of a broad selection of references to extra materials. Readers can subscribe to their newsletter to receive the latest news directly in their emails.

BitMEX Research

BitMEX is among the top crypto exchanges, but what many do not know is that their research blog is also one of the most in-depth in the crypto space. The BitMEX journalists flame their posts in a financial journal style, covering everything related to blockchain technology and crypto markets.
The BitMEX team strives to offer deep insights into why and how one or more current happenings are influencing the crypto world in a certain way. A good example of this would be their efforts in covering stablecoins, which ends with what current stablecoin projects are trustworthy and why, with the support of financial indicators. Because of these factors and many more, it is easy to group their blog among the top crypto blogs.

Bitcoin Magazine

Bitcoin Magazine is the world’s first magazine and one of the most established sources of Bitcoin news,  information, market analysis and expert commentary on Bitcoin, its underlying technology, and the entire industry that has evolved around it. The magazine is owned by BTC Media LLC, the media and publishing subsidiary of BTC Inc. Since 2012, it has offered detailed analysis, research, education, and thought leadership at the intersection of finance and technology.
The publication strives to offer accurate, timely, and relevant content to the crypto community.
Specifically, Bitcoin Magazine is passionate about Bitcoin and is keen to share news about individuals, businesses, and technological advancements that are transforming the world. Whether you are blockchain newcomer, savvy developer, investor, or entrepreneur, the platform aims to inform and educate you with quality content that meets the international editorial and journalistic standards, featuring both the highs and the lows of the crypto industry.

BitcoinTechWeekly

BitcoinTechWeekly differs a lot from the rest of this pack of websites for one main reason. It is categorically geared toward those who are not only seeking Bitcoin news but already have a technical understanding of blockchain technology.
Most of their posts focus on the latest software development tools, technical analysis and network updates in the industry, making it the best choice for technically savvy blockchain individuals.
Since they purposely style themselves as covering “weekly bitcoin tech news,” it is likely that their opinions will be subject to the Bitcoin maximalist school of thought.

A16z Blog

Marc Andreessen is one of the founders of Mosaic and Netscape, which ended up being the world’s first widely adopted web browsers. Then, in 2009 together with Horowitz, he established the Andreessen Horowitz Venture Capital Company, which runs the A16z Blog.
While this venture capital firm invests in both early-stage start-ups and established companies, its cryptocurrency blog is unique in two main ways.
First, as one of the Bitcoin large scale investor pioneers, they speak well from the institutional angle of the crypto industry. Second, the A16z Blog does not shy to share market insights including critical information related to cryptocurrencies and blockchain, to truly offer one of the most balanced crypto blogs in existence.

Jimmy Song Blog

Like Vitalik Buterin, Jimmy Song was one of the pioneers of all things blockchain. Categorically, he was one of the first developers to work on the Bitcoin blockchain. In his Medium blog, he draws on his opinions and the knowledge gained from his involvement in blockchain projects, to provide a professional analysis on topics related to the blockchain space.
In some of the articles, his explanations are specifically meant for developers, to connect a blockchain idea to something related to programming. In other guides, he concentrates on explaining concepts, such as the real potential of Asset-Backed Tokens (ABTs) to the general audience. Generally, Song’s publications attain the right balance regarding the audience it caters to and the content they cover.

Multicoin Capital

For those who are searching for crypto hedge funds, then look no further. The Multicoin Capital blog regularly curates what it terms as the essential podcast about the blockchain industry crypto markets.
They mainly cover how to excel as a cryptocurrency investor using more traditional investing approaches, such as achieving the ideal alpha. In other words, they strive to inform both new and experienced cryptocurrency investors on how to beat the crypto waves. Due to this, it is considered as one of the best crypto blogs available.

Conclusion on the list of the best crypto blogs

By including these blogs in your bookmark, you will always be updated with blockchain, financial markets and cryptocurrency news from all over the world. These blogs and personalities were critically chosen since they have already established themselves as reliable and respected sources of information.
Remember, public sources can guide you only by analyzing data from different sources, questioning everything, and suspecting manipulation under every analysis of “unique” information and statistical facts.