Clik here to view.
Encoding the Hive query file in Azure HDInsight
Today at Microsoft we were using Azure Data Factory to run Hive Activities in Azure HDInsight on a schedule. Things were working fine for a while, but then we got an error that was hard to understand....
View ArticleClik here to view.
Rerunning many slices and activities in Azure Data Factory
Today someone asked me how to run all the data slices in their data factory on-demand in an ad-hoc fashion, to say run the whole pipeline again from scratch. For example, if you have a one-time copy...
View ArticleClik here to view.
Capture Microsoft Azure Stream Analytics logs
Microsoft Azure Stream Analytics makes building real time solution very easy. Developers can build Stream Analytics job with few clicks. While running Stream Analytics jobs, you may encounter an error...
View ArticleClik here to view.
Make a custom role for users to Create an Azure Data Factory
Update – a fix is coming soon so that you do not need to use this workaround. Hope we can get Data Factory Contributors to be the minimum permissions to create a Data Factory from the Azure portal....
View ArticleClik here to view.
HDFS gets full in Azure HDInsight with many Hive temporary files
Sometimes when Hive is using temporary files, and a VM is restarted in an HDInsight cluster in Microsoft Azure, then those files can become orphaned and consume space. In Azure HDInsight, those temp...
View ArticleClik here to view.
How to Find and Kill a running Yarn Application Master in HDInsight with and...
Today we faced a challenge in HDInsight not knowing the SSH user password to terminal into the server, and we needed to kill some running Hive jobs that were too far gone and taking too many resources....
View ArticleClik here to view.
How to Lock a Resource Group to prevent accidental deletion of resources like...
Did you know it is possible to prevent accidental deletion of resources in Azure? This could apply to any number of resource, HDInsight, Stream Analytics jobs, Data Factories, DocumentDB accounts, etc....
View ArticleClik here to view.
HDInsight Name Node can stay in Safe mode after a Scale Down
This week we worked on an HDInsight cluster where the Name Node has gone into Safe mode and didn’t leave that mode on its own. It’s not very common, but I wanted to share why it happened, and how to...
View ArticleHDInsight Hive Metastore fails when the database name has dashes or hyphens
Working in Azure HDInsight support today, we see a failure when trying to run a Hive query on a freshly created HDInsight cluster. Its brand new and fails on the first try, so what could be wrong? Our...
View ArticleClik here to view.
How to call a Azure Machine Learning Web Service from NodeJS
Azure machine learning allows data scientists and developers to embed predictive analytics into applications. To learn more about Azure machine learning visit Azure machine learning documentation . A...
View ArticleClik here to view.
Encoding 101 – Exporting from SQL Server into flat files, to create a Hive...
Today in Microsoft Big Data Support we faced the issue of how to correctly move Unicode data from SQL Server into Hive via flat text files. The main issue faced was encoding special Unicode characters...
View ArticleClik here to view.
Encoding the Hive query file in Azure HDInsight
Today at Microsoft we were using Azure Data Factory to run Hive Activities in Azure HDInsight on a schedule. Things were working fine for a while, but then we got an error that was hard to understand....
View ArticleClik here to view.
Incremental data load from Azure Table Storage to Azure SQL using Azure Data...
Azure Data Factory is a cloud based data integration service. The service not only helps to move data between cloud services but also helps to move data from/to on-premises. For example, moving data...
View ArticleClik here to view.
How to allow Spark to access Microsoft SQL Server
Today we will look at configuring Spark to access Microsoft SQL Server through JDBC. On HDInsight the Microsoft SQL Server JDBC jar is already installed. On Linux the path is...
View ArticleClik here to view.
Multi-Stream support in SCP.NET Storm Topology
Streams are in the core of Apache Storm. In most cases topologies are based on a single input stream, however there are situations when one may need to start the topology with two or more input steams....
View ArticleClik here to view.
Using Azure SDK for Python
Python is a great scripting tool with a large user base. In a recent support case I needed a way to constantly generate files with some random data in windows azure storage (wasb) in order to process...
View ArticleClik here to view.
A KMeans example for Spark MLlib on HDInsight
Today we will take a look at Sparks’s module for MLlib or its built-in machine learning library Sparks MLlib Guide . KMeans is a popular clustering method. Clustering methods are used when there is...
View ArticleClik here to view.
HDInsight Hive Metastore fails when the database name has dashes or hyphens
Working in Azure HDInsight support today, we see a failure when trying to run a Hive query on a freshly created HDInsight cluster. Its brand new and fails on the first try, so what could be wrong?Our...
View ArticleClik here to view.
How to call a Azure Machine Learning Web Service from NodeJS
Azure machine learning allows data scientists and developers to embed predictive analytics into applications. To learn more about Azure machine learning visit Azure machine learning documentation . A...
View ArticleClik here to view.
Encoding 101 - Exporting from SQL Server into flat files, to create a Hive...
Today in Microsoft Big Data Support we faced the issue of how to correctly move Unicode data from SQL Server into Hive via flat text files. The main issue faced was encoding special Unicode characters...
View Article
![[TVRIP] Vertex Force [バーテックスフォース] 第01話](http://animerss.com/img/TVRIP-Vertex-Force.jpg)
