I'm in charge of Azure Functions kafka Extension, but recently I have a lot of pull requests, so I want to write E2E tests in languages other than C #. It has become. Most people change things just by looking at C #, but from my point of view, does it work in other languages? It's annoying to test every time, so I decided to create a mechanism.

Challenges when building the current package and loading it into a package in another language

The original Extension of this project was written in C #. So, using the created NuGet package, it feels like other languages will work, so the following is necessary.

On Linux, create a package from the current source.
Load the package into the csproj file used by the package such as Java and Python.
Start the server and test

At first glance it looks like nothing, but there are some considerations

What to do with the version to build? If you package it obediently with the current project version, it will be packaged with the current version. In that case, if the current version is published from NuGet source, it may be referenced, so False Positve is likely to occur.
What version should be loaded from the Java and Python packages? It's tedious to change it manually every time the version goes up, so I don't want to change it.
How to load the package? It seems that csproj cannot read the package by specifying the file.

After all, what happened was 100.100.100, well, I wouldn't update that much. I set up a version like that, and made it a simple solution to do so with the source code. There seems to be a better one.

Build C # package

Assuming dotnet is already installed, I wrote and ran the following command on WSL2. The important point is dotnet pack -o temp --include-symbols src / Microsoft.Azure.WebJobs.Extensions.Kafka / Microsoft.Azure.WebJobs.Extensions.Kafka.csproj /p:Version=100.100.100 Package Specify the version 100.100.100, execute pack, and output to the temp directory. After that, it is copied to the directory for each language. The directories for each language are configured to be packed into Docker.

#!/bin/bash

CURRENT_DIR=`pwd`
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd $DIR/..

WORKING_DIR=temp
if [ -d "$WORKING_DIR" ]; then rm -rf $WORKING_DIR; fi

dotnet pack -o temp --include-symbols src/Microsoft.Azure.WebJobs.Extensions.Kafka/Microsoft.Azure.WebJobs.Extensions.Kafka.csproj /p:Version=100.100.100


PACKAGE_DIR=test/Microsoft.Azure.WebJobs.Extensions.Kafka.LangEndToEndTests/server/java8/packages
if [ -d "$PACKAGE_DIR" ]; then rm -rf $PACKAGE_DIR; fi
mkdir $PACKAGE_DIR
cp temp/* $PACKAGE_DIR

cd $CURRENT_DIR

Specifying NuGet Source

How can I get a NuGet file to read? I thought, I set Local Source and made it read. Currently, there was a command called dotnet nuget add source, so I tried using it. The old information was that I installed mono and used nuget.exe, but now it seems unnecessary.

dotnet nuget add source

RUN dotnet nuget add source /workspace/packages/

By the way, issuing this command updates the default NuGet.Config as follows: At first, I thought about changing the version fixing method and the priority of reading nuget source so that Local always reads first, but after all False Positive is worrisome, so fixing the version I made it.

# pwd
/root/.nuget/NuGet
# cat NuGet.Config 
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  <add key="Package source 1" value="/workspace/packages/" />
</packageSources>

Java, Python side csproj file

Because of the specification, the csproj file on the Java and Python side is only the version 100.100.100.100. This will automatically include a build of the current source code.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
	<WarningsAsErrors></WarningsAsErrors>
	<DefaultItemExcludes>**</DefaultItemExcludes>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Kafka" Version="100.100.100" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.1.7" />
  </ItemGroup>
  <ItemGroup>
    <None Update="confluent_cloud_cacert.pem">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
  </ItemGroup>
</Project>

Dockerfile

The final Dockerfile looks like this: Easy and messy.

FROM mcr.microsoft.com/azure-functions/java:3.0-java8-core-tools

# Copy Local File
RUN mkdir /workspace
COPY . /workspace
WORKDIR /workspace
# dotnet 

ENV LD_LIBRARY_PATH=/workspace/target/azure-functions/kafka-function-20190419163130420/bin/runtimes/linux-x64/native

RUN dotnet nuget add source /workspace/packages/

RUN mvn clean package

ENTRYPOINT [ "/usr/bin/mvn", "azure-functions:run" ]

Summary

If you're a C # master, you might come up with a better way. Please let me know if you have any. After this, I was able to start and work with the Kafka server with docker-compose, so after that, I wrote the E2E framework in C # and incorporated it into Azure DevOps, and I'm done.

Supplement 11/9/2020

I got good advice from Mr. Shibayan, so I decided to use the package reference. You don't have to make a Local Nuget anymore, so this one looks better. Since I pack it in Docker, I thought that PackageReference could not be used, but it is simple and I can make the context of docker build the project root. However, I will leave the entry because it was a good learning of the command.

extension.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
	<WarningsAsErrors></WarningsAsErrors>
	<DefaultItemExcludes>**</DefaultItemExcludes>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.1.7" />
  </ItemGroup>
  <ItemGroup>
    <None Update="confluent_cloud_cacert.pem">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\..\..\..\src\Microsoft.Azure.WebJobs.Extensions.Kafka\Microsoft.Azure.WebJobs.Extensions.Kafka.csproj" />
  </ItemGroup>
</Project>

Dockerfile

FROM mcr.microsoft.com/azure-functions/java:3.0-java8-core-tools

# Copy Local File
RUN mkdir /workspace
COPY . /workspace
WORKDIR /workspace/test/Microsoft.Azure.WebJobs.Extensions.Kafka.LangEndToEndTests/server/java8
# dotnet 

ENV LD_LIBRARY_PATH=/workspace/target/azure-functions/kafka-function-20190419163130420/bin/runtimes/linux-x64/native

RUN mvn clean package

ENTRYPOINT [ "/usr/bin/mvn", "azure-functions:run" ]

docker build -t testj8 -f test/Microsoft.Azure.WebJobs.Extensions.Kafka.LangEndToEndTests/server/java8/Dockerfile .

Resource