v3.2.6 Argo UI failed to render pod log

Summary
What happened/what you expected to happen?

What version of Argo Workflows are you running?
3.2.6 (we just upgrade it from 3.2.3 today, so I guess this issue might be introduced between 3.2.3 and 3.2.6. We'll try to roll back to 3.2.3 and see if this issue still exist)
Diagnostics
I'm still not able to find a minimal reproducing example, since this issue only happened in some of our pipelines, and not sure which part of the template triggered this issue.

Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

Most likely introduced by this commit:
f5f6899f5 fix: Add pod name format annotation. Fixes #6962 and #6989 (#6982)
@JPZ13 would you be able to look into this please?

Most likely introduced by this commit:
f5f6899 fix: Add pod name format annotation. Fixes #6962 and #6989 (#6982)
@JPZ13 would you be able to look into this please?

On it

@liuzqt Can you share what the pod names are in kubernetes for this workflow that are resulting in the error on the UI? I'm working on a fix and need to make sure there's a 1:1 match between the pod names as they appear in k8s and what our UI pod name function returns
Thanks for reporting the issue!

@JPZ13  the Pod UI is not working, I tried to find their information from the json meta, hopefully this can help:
"patch-processing-pipeline-ksp78-1623891970": {
                "id": "patch-processing-pipeline-ksp78-1623891970",
                "name": "patch-processing-pipeline-ksp78.retriable-map-authoring-initializer",
                "displayName": "retriable-map-authoring-initializer",
                "type": "Pod",
                "templateScope": "local/",
                "phase": "Succeeded",
                "boundaryID": "patch-processing-pipeline-ksp78",
                "startedAt": "2022-01-20T01:45:35Z",
                "finishedAt": "2022-01-20T01:45:39Z",
                "progress": "1/1",
                "resourcesDuration": {
                    "cpu": 3,
                    "memory": 3
                },
                ...

"patch-processing-pipeline-ksp78-1769534216": {
                "id": "patch-processing-pipeline-ksp78-1769534216",
                "name": "patch-processing-pipeline-ksp78.NODE-RETRIABLE-MAP-AUTHORING(1).NODE-GENERATE-EMLANE.pre-emlane-map-builder",
                "displayName": "pre-emlane-map-builder",
                "type": "Pod",
                "templateRef": {
                    "name": "map-builder-task",
                    "template": "map-builder"
                },
                "templateScope": "namespaced/node-generate-emlane-task",
                "phase": "Failed",
                "boundaryID": "patch-processing-pipeline-ksp78-242024017",
                "message": "Error (exit code 1)",
                "startedAt": "2022-01-20T01:53:05Z",
                "finishedAt": "2022-01-20T01:53:08Z",
                "progress": "1/1",
                "resourcesDuration": {
                    "cpu": 4,
                    "memory": 4
                },
                ...

That is super helpful @liuzqt. Thank you for the extra info. I'll tag you on the PR for the fix once it's ready

@JPZ13  sorry actually I'm just an Argo user in our company and I have to ask our Argo team to help verify this fix later. I'll update once I get feedback from them.

@JPZ13 sorry actually I'm just an Argo user in our company and I have to ask our Argo team to help verify this fix later. I'll update once I get feedback from them.

Sounds good. Thank you @liuzqt

@JPZ13 I am in touch of @liuzqt’s team. Let me know if there’s any other information you need.


remove use module::{self}, less unneeded allocations, a bit of clippy…

… fixes

no thx, some suggestions were good (like use of format! and removing {self}) but otherwise I don't want these changes in my code


Fix loading info and zoom label

Show loading data info when ticking for new network layout #176;
Keep node label size fixed when zooming #177;
Fix minor issues for polygon selection.

LGTM


Slash Commands Get Endpoint

A new endpoint was added to retrieve a single slash command
https://github.com/discord/discord-api-docs/commit/093ea60e52e86d78ae22635abf5b64c358e6d04d

I already added these functions in #444


Error on startup

Description
After updating the plugin via PackerSync a few minutes ago, to the latest version, the change in plugin/lspsaga.lua#74 (line 74) causes an
error on startup for me.
The change has been introduced only a few hours ago.
E5113: Error while calling lua chunk: ...m/site/pack/packer/start/lspsaga.nvim/plugin/ls
psaga.lua:74: invalid key:
stack traceback:
        [C]: in function 'nvim_set_hl'
        ...m/site/pack/packer/start/lspsaga.nvim/plugin/lspsaga.lua:74: in main chunk

Expected Behavior
Startup works without an error.
Actual Behavior
Starting up Neovim causes an error.

You my man, are fast. Fixed it


Hello, can we add deep reinforcement learning to optimize this system

Hello, can we add deep reinforcement learning to optimize this system

Hello,
Yes, you can achieve this by extracting the relevant state information from the Simulation module (see Metrics.py for an example) to feed to the agent. To apply the agent's decisions, you can use SystemEvents from the Manager module to reconfigure the relevant parameters (you can look at some existing reconfiguration logic in Manager.py). However, this repo contains no built-in RL algorithms, so you must implement and integrate them into the simulation yourself.
I hope that helps! Feel free to open a pull request if you go through with this and want to share.

Thank you very much

No problem!


False positives for type equality

Unfortunately, while #33 now catches deep anys, it seems to have weakened other equality checks. In 0.7.3, this assertion would (correctly) fail:
type NoParam = () => void;
type HasParam = (s: string) => void;

expectTypeOf<NoParam>().toEqualTypeOf<HasParam>();

But starting with 0.7.4, which contains that fix, this assertion passes.

Thanks @dfreeman, good catch. This is a bug - although it's often a good idea to narrow down assertions on functions with .parameters, .returns etc. e.g.
type NoParam = () => void
type HasParam = (s: string) => void

expectTypeOf<NoParam>().parameters.toEqualTypeOf<[]>()
expectTypeOf<NoParam>().returns.toEqualTypeOf<void>()

expectTypeOf<HasParam>().parameters.toEqualTypeOf<[string]>()
expectTypeOf<HasParam>().returns.toEqualTypeOf<void>()

This isn't always possible though, so will find a fix for the toEqualTypeOf regression.


feat(tier4_localization_rviz_plugin): add visualization of pose with covariance history

Description
add plugin to visualize  covariance estimated by NDT scan matching continuously on Rviz.

Related links
None.
How was this PR tested?
Run Autoware in my laptop.
Notes for reviewers
None.
Interface changes
None.
Effects on system behavior
None.

@YamatoAndo The maintainer was updated with https://github.com/autowarefoundation/autoware.universe/pull/8184. After the code owner update, would you please review this PR?

@isamu-takagi Sure!

Even though the path checkbox is turned off, the "Arrow" is not being removed.


Add CRLF character to end of multipart body.

Per RFC 1521 this character is required at the end of multipart body.
Per RFC 2046 there is an optional CRLF character at the end of the multipart-body to delimit between the body and optional epilogue.
Some server implementations break if this character is not added, so adding it here as it complies with both RFCs.

Awesome! Thanks!

@ixti - np! I've worked around this for the time being, but any idea of when it would be included in a new release?

@mhickman I'll do that ASAP. Latest - this weekend :D

Thanks!


Add a new right-aligned "description" property to TreeItems

Similar to the timestamp in the Timeline view, it would be nice to have the ability to have a string that would be rendered right-aligned for each tree item.
Maybe something like:
export class TreeItem {
    label?: string;
    description?: string | boolean;
    ...
    endDescription?: string; // or endDetails? or even just details, but that feels wrong given the other usages of details
}

@misolori This could also be used for a checkmark for pr reviews

We should first discuss this in the UI sync.
The file decoration API is also being finalized soon. We can you that for a checkmark for PR reviews too.

Current:

Example where we hide the right-aligned description on hover when the action is shown:

Example in the panel:

Seems like a fair request to me, but happy to discuss in a UX call.

@bpasero we actually did discuss this in last week's ux sync, seemed like everyone was in agreement of this but the specific detail of whether we hide the description label was under discussion. I personally like hiding the description on hover as that is secondary information and actions have a higher priority.

Since we are trying not to add new proposed API in October, moving to November.

@alexr00 Any  updates on this proposal? Would also love to have this in my vscode extension. I would also hide it by default, but you can add a second property to control the behavior (auto-hide, show, show-fully).

This would be helpful for me. I'm the developer of Godot Tools, that provides support for the Godot Game Engine.
One of my features is a TreeView that shows a preview of an open scene file:

This shows the hierarchical view of the Nodes that make up the scene. You can see I'm currently using FileDecorations to provide icons on the right side of the TreeItems. Unfortunately, I can only have two active Decorations on any one TreeItem, but a Node has more relevant properties that would be useful to display.
Here's the Godot Editor's scene tree viewer with 5 different icons I'd like to be able to display:

Having a right aligned description would give me another option to display symbols that don't require interaction but should still be visible all the time.
Additionally, the FileDecorations API is very awkward for this, because I know 100% of the information about each TreeItem when I first make it. Having to parse the scene file, build the TreeItems, refresh the TreeView, and then still having to trigger each DecorationProvider for every TreeItem is convoluted, annoying, and wasteful.


Penv 527 heavy version check

- What I did
penv-527 version check on heavy

platform updated
added gbe version, unit tests
added version check in grpc mock with tests
added GBE shutdown function
added shutdown on vesrion error

- How I did it
- How to verify it
- Description for the changelog

Codecov Report

Merging #145 into master will increase coverage by 0.06%.
The diff coverage is 77.02%.

@@            Coverage Diff             @@
##           master     #145      +/-   ##
==========================================
+ Coverage   74.42%   74.49%   +0.06%
==========================================
  Files          48       48
  Lines        3398     3466      +68
==========================================
+ Hits         2529     2582      +53
- Misses        720      733      +13
- Partials      149      151       +2

Impacted Files
Coverage Δ

testutils/betestsetup/betestsetup.go
46.51% <0.00%> (-2.33%)
:arrow_down:

testutils/grpc.go
76.47% <69.81%> (-23.53%)
:arrow_down:

etl/extractor/platform_impl.go
79.85% <100.00%> (+4.63%)
:arrow_up:

testutils/connectionmanager/connectionmanager.go
95.34% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bbc86b1...a300c25. Read the comment docs.


scipy.ncx2.sf should be monotone decreasing

I'm getting unexpected behavior from scipy.ncx2.sf.  The survival function should eventually go to zero for large enough inputs, but instead I'm seeing a kind of "foot", i.e., it is bottoming out at a positive value.  The following interactive I/O demonstrates the issue.
Reproducing code example:
(Pdb) v_T_norm[::50]
array([  4.99515382e+00,   1.07617327e+01,   2.31854502e+01,
4.99515382e+01,   1.07617327e+02,   2.31854502e+02,
4.99515382e+02,   1.07617327e+03,   2.31854502e+03,
4.99515382e+03,   1.07617327e+04,   2.31854502e+04,
4.99515382e+04])
(Pdb) nu
20.0
(Pdb) lam
499.51538166556196
(Pdb) ncx2.sf(v_T_norm[::50], df=nu, nc=lam)
array([  1.00000000e+00,   1.00000000e+00,   1.00000000e+00,
1.00000000e+00,   1.00000000e+00,   1.00000000e+00,
6.64666099e-01,   4.16325934e-05,   4.16325934e-05,
4.16325934e-05,   4.16325934e-05,   4.16325934e-05,
4.16325934e-05])

After a quick look, I can see that the problem is in scipy.special.chndtr.  The implementation of nxc2 doesn't define a survival function, so it inherits the default implementation, which is simply 1 minus the CDF.  The CDF for ncx2 ultimately calls scipy.special.chndtr, and it exhibits the "plateau" behavior:
In [32]: from scipy.special import chndtr

In [33]: nu = 20.0

In [34]: lam = 499.5

In [35]: chndtr([500, 750, 1000, 2000, 4000], nu, lam)
Out[35]: array([ 0.33945802,  0.99995706,  0.99995841,  0.99995841,  0.99995841])

The next step is to figure out what is wrong with chndtr.

The problem is in the implementation of cumchn in the CDFLIB Fortran code.  I'll have a pull request with a fix once I get some known accurate values to use in unit tests.

A question on stackoverflow also notes the problem with the accuracy of ncx2.cdf.  With the change made in https://github.com/scipy/scipy/pull/8673, that example gives
In [4]: ncx2.cdf([1200, 1500, 2000, 5000], 2, 1000).tolist()
Out[4]:
[0.9986639334269182,
 0.9999999999993727,
 1.0000000000000306,
 1.0000000000000306]

The values are more accurate, but (as noted in https://github.com/scipy/scipy/pull/8673), the values still "plateau" at a value that is not 1, and in some cases the plateau value is greater than 1.

The newer issue should be deleted.
On Thu, Apr 12, 2018 at 12:22 AM, pyguy3<EMAIL_ADDRESS>wrote:

Just opened #8712 https://github.com/scipy/scipy/issues/8712 which is
the same issue. Please let me know if I should delete that one. My issue
report also mentions an alternative algorithm that could be used.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/scipy/scipy/issues/8665#issuecomment-380704486, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABBBxEQaYwLZZO-_rvryM-gWM6EZQ4fZks5tnwC7gaJpZM4TEPIM
.

Just wanted to add that this seems to be not only an accuracy problem but it has significant issues.
from scipy.stats import ncx2
import numpy as np
import matplotlib.pyplot as plt
from rpy2.robjects.packages import importr  # for comparison
stats = importr(str('stats'))
x = np.arange(5000, 15000, 1)
scdf = [ncx2.cdf(i,30,i) for i in x]
rcdf = [stats.pchisq(i,30,i)[0] for i in x]
f, ax = plt.subplots(figsize = (14, 10))
ax.plot(x, scdf, c='b', label='from scipy')
ax.plot(x, rcdf, c='r', label='from rpy2')
ax.legend(prop={'size': 30})

results:

A change that was implemented last year moved the point at which the bad behavior begins out to the right, but did not eliminate the problem.

The fix above proposed for #11474 , seems to solve this issue.

This looks perfect!


nomis: DSOS: allow more S3 get permissions

We're hitting a permission with S3 permissions in nomis - simplifying the S3:Get permissions.

Don't think this will fix issue, raised ticket with AWS


[QA]  PromQL subtraction operator  is not right?

What did you do?
I use PromQL to check my service code,  I just want to known whether there is 4xx expect 400 / 419 case.
Query like screenshot:

What did you expect to see?

no  value smaller than 0.

What did you see instead? Under which circumstances?

some value smaller than 0

Environment

System information:

Ubuntu 14.04

Prometheus version:

v 1.4.1

I known there is a better query for my case , but I want to known  what's the problem for my current query.

That looks like floating point inaccuracy.


docs: broken link at https://xarray.pydata.org/en/stable/user-guide/terminology.html

What happened?
The link to https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_data_set_components.html#variables in the "Variables" heading is broken, I think it should now link to the NetCDF user guide: https://docs.unidata.ucar.edu/nug/current/netcdf_data_set_components.html#variables
What did you expect to happen?
Link works
Minimal Complete Verifiable Example
1. Click on the  "NetCDF-like variable" link on page https://xarray.pydata.org/en/stable/user-guide/terminology.html.
2. 404

MVCE confirmation

[X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
[X] Complete example — the example is self-contained, including all data and the text of any traceback.
[X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
[X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output
N.A.

Anything else we need to know?
No response
Environment

N.A.

Thanks @phil-blain This should be a really easy fix if you're up for contributing: See https://github.com/pydata/xarray/blob/main/doc/user-guide/terminology.rst for the source and https://docs.xarray.dev/en/stable/contributing.html#contributing-to-the-documentation for instructions

Yeah, I just wanted to confirm this was indeed the intended link target:)

Closing, I forgot the "Closes" trailer when I authored my commit and PR.


Master.ServiceType flag for Spark Master is only affects Spark webUI service

Is this a request for help?:
no

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Version of Helm and Kubernetes:
helm
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

Kubernetes
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
Spark
What happened:
Master.ServiceType flag for Spark Master is only affects Spark webUI service
What you expected to happen:
Master.ServiceType should set type of spark master service here (https://github.com/helm/charts/blob/master/stable/spark/templates/spark-master-deployment.yaml#L15) not the type of web ui service https://github.com/helm/charts/blob/master/stable/spark/templates/spark-master-deployment.yaml#L32 . There should be separate service type value for webUI just like Zeppelin flags
How to reproduce it (as minimally and precisely as possible):
helm install --set Master.ServiceType=NodePort will not change service type of master service.
Anything else we need to know:

Because master was using ClusterIP as service type by default, I was not able to submit a spark job from outside of cluster


how to use it using C# with .NET 4.5 framework

could you give me example or simple code allow me to use it using .NET 4.5 framework because nothing work with me , but on core it work fine
thank you ,,

Hello! Can you describe an issue?
If you get something like
"FileLoadException: Could not load file or assembly 'Newtonsoft.Json, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)"
try to install Newtonsoft.Json package (version 10.0.*)

I have already installed all the libraries and Run without errors .. but when I request a translation , nothing happens and the application screen freezes without any messages or any response .. Maybe it has to do with async and wait functions and 4.5 framework.

thank ,, i found my problem here
https://stackoverflow.com/questions/36925852/c-sharp-getstringasync-never-returns-proper-way-of-doing-async-get


[Feature Request] LZ4 compression

Hi,
I'm developing (as a hobby) a Tile-based 2D platformer game in Unity. At the moment I'm still creating some parts for a custom level editor. One of the more important issues in this project is data (de-)serialization:
I have a Level class which derives from ScriptableObject (in order to be able to edit certain values in the Unity Inspector). This class holds a Tile[,] array which gets serialized to a byte[] array / deserialized from it.
Currently I'm using neuecc's MessagePack serializer but if your benchmarks are still up to date I'd like to try out Ceras since I'm looking for maximum performance.
However, one thing Messagepack has is LZ4 compression. Since the byte[] array in my Level class needs to be serialized again using the default [SerializeField] attribute for the Unity serializer, reducing its size by applying LZ4 compression improves editor responsiveness immensely. Without compression I've even had the Unity editor freeze up/crash on me when the ScriptableObject of a large level is selected in the Project view (since apparently the Inspector deserializes the SO every frame to draw the fields, possibly applies changes and then serializes it again).
I saw that LZ4 is already on your list of planned features. Since I didn't find a schedule for these planned features my hope is that opening a feature request for this will help increase its priority ;)
My proposal would be to have an overload of your Serialize/Deserialize method with an optional parameter for the compression type. This way existing users can keep their syntax as is while adding something like "CompressionType.LZ4" as additional parameter applies the compression.
This would also allow changing it to "CompressionType.GZip" later (although GZip is not a priority for me since even with the smaller size resulting from it the compression would usually take longer than LZ4).
The default value could be "CompressionType.None" so that anyone who would need to could keep calling the same method overload and simply pass a variable with the appropriate CompressionType for their purposes.
If you need any more info let me know.
Cheers!

Hi, should be relatively straightforward to add this.
My proposal would be to have an overload of your Serialize/Deserialize method with an optional parameter for the compression type. This way existing users can keep their syntax as is while adding something like "CompressionType.LZ4" as additional parameter applies the compression.
Hmm, normally I'd agree, but I think a normal setting would be better.
Parameters can change, settings can not. That enables potential optimizations to be done because every feature/setting can rely on settings never changing (that has happened multiple times during development already)
Also, adding more parameters/overloads makes using it slightly harder for beginners (even if optional), and with the recent addition of "Advanced.SerializeStatic()" I'd have to replicate every api change into yet another set of methods.
Currently I'm using neuecc's MessagePack serializer but if your benchmarks are still up to date I'd like to try out Ceras since I'm looking for maximum performance.
Since I'm currently pretty busy, may I ask you to first test Ceras with a (custom compression layer on top if needed) to make sure that it even works for you at all? (in terms of features, speed, output size...)
I'd like to avoid spending (probably a considerable amount of) time if it doesn't help anyone in the end.
That being said, if I'll add it, then I'll probably implement the full feature all at once, so not just LZ4 but also GZip, and maybe also using ZLib directly to make use of sync points (which would be awesome for networking scenarios).

Oh, of course, setting the variable for the compression in a Settings class would also work just fine.
I'll need to tie up a few loose ends in my code before I'll be able to replace the MessagePack serializer with Ceras but I'm pretty confident that it should work without any problems. If you'd like me to confirm this by testing it I might be able to do this within the next few days depending on my own schedule.
Since this is a hobby project switching out the serializer is not a time-critical factor for me. So if you are busy I completely understand if implementing compression might take a while. I should have some other parts in my code to which I could tend in the meantime ;)

If you'd like me to confirm this by testing it I might be able to do this within the next few days depending on my own schedule.
Yes, that'd be great.
While doing preliminary testing you'll probably want to use some external LZ4 library (or maybe using messagepacks built in LZ4 encoder) to ensure that the total time of serialization+compression / deserialization+decompression is within your acceptable range.
If/When you tell me what you can see switching to Ceras would actually be beneficial for your project I'll make time to get this feature implemented asap. It will probably take just 1~2 days from then on.
Btw: there are a lot of settings and things that have a noticeable impact on Ceras' serialization performance. Most notably that's using the right overload (the one where you give a buffer by reference, and get the amount of bytes written as a return value; instead of the one where you only get the result-byte-array) and turning off reference-serialization (so Ceras works like almost every other serializer, like msgpack or newtonsoft json; namely not taking care of circular-references and de-duplication). There are a few more things that have a minor impact on performance - like using KnownTypes - but those should be obvious from reading the intellisense documentation (if you ever want to investigate that deeply).
If you have any questions or problems don't hesitate to ask! (open a separate issue, or join the discord, or whatever :P) If there are any issues, we can most likely fix them pretty easily. :+1:

Okay, I've fixed a few things now in the last few days in preparation for testing.
I've downloaded Ceras from here: https://ci.appveyor.com/project/rikimaru0345/ceras/build/artifacts ("ceras_net45.zip" version), extracted it and put the contents into "Assets/Plugins/Ceras/" in my Unity project.
Then in my Level class I changed my "Save" method to look like this:
public void Save()
{
	SerializerConfig config = new SerializerConfig();
	config.DefaultTargets = TargetMember.AllFields;
	CerasSerializer serializer = new CerasSerializer(config);

	using(new StopwatchScope("Saving Level"))
	{
		int count = serializer.Serialize<Tile[,]>(tiles, ref serializedTiles);
		Debug.Log("serialized: " + count);
	}
}

Unfortunately calling this method results in this error message:

ArgumentException: Expression of type 'Tile[,]' cannot be used for parameter of type 'Tile[]' of method 'Void Serialize(Byte[] ByRef, Int32 ByRef, Tile[])'
Parameter name: arg2

I think this is trying to indicate that a 2D array gets wrapped to a 1D array internally but there's some problem occuring?

Hey, currently Ceras doesn't support multi-dimensional arrays, but I think I can easily fix it.
I'll take a closer look at this this evening or tomorrow evening.

@Hurri04
Can you take a look at the latest commit? 7a3573cdb4174f1ed543f5dbf1c272ce8a44e867
I've added support for multidimensional arrays. :+1: :smile:
By the way: the timing in your Save method will probably report a slower than expected time because most of the initialization happens in the first serialization. Ceras generates code for your types then (and they're only known once you actually use them in a Serialization/Deserialization). Skip the first measurement to get a more accurate result.

I'm having a bit of trouble compiling Ceras myself.
After using [Build]->[Build Ceras] in Visual Studio I now have these files in this folder:
"...\Workspace\C#\Ceras\src\Ceras\obj\Release\net45"

However, copying them into "Assets/Plugins/Ceras" folder in my project results in Unity showing this error message:

Assembly 'Assets/Plugins/Ceras/Ceras.dll' will not be loaded due to errors:
Unable to resolve reference 'System.Buffers'. Is the assembly missing or incompatible with the current platform?
Unable to resolve reference 'System.Runtime.CompilerServices.Unsafe'. Is the assembly missing or incompatible with the current platform?

For unity, try using the output in: \bin\Release\netstandard2.0
I will try to add LZ4 compression in the next days and then publish a compiled version that you can download from appveyor as well :+1:
Not sure yet if I'll have compression as a core feature, or if I'll put it into a separate .dll like with the Unity extensions or Immutable Collections package.

\bin\Release\netstandard2.0 contains these files:

But using these results in the same error.
Are there any settings I'd need to set / packages I'd need to install for this to compile properly?

Since Unity supports .net standard 2.0 it is strange that it has trouble loading Buffers and Unsafe.
Take a look at all the debug/release and different platforms, you should be able to find System.Buffers.dll and System.Runtime.CompilerServices.Unsafe.dll.
Copy them to your Unity folder along with the Ceras.dll.
I think the issue Unity has is that those are packages that are downloaded through nuget.
Visual Studio does it for you, automatically, but Unity does not respect the Ceras.deps.json, and doesn't download the needed packages. So that's why you need to do that step manually it seems.
From what I've heard Unity is aware of the issue and there are things planned to at least make this easier / more obvious in future Unity versions.

I might be going slightly mad...
Either I've been looking in the wrong folders yesterday or their contents have been modified since then.
\bin\Release\net45 now contains these files:

and they work.
I've done a few test runs of saving and loading an (empty) Level off 700x400 Tiles (which is the biggest Level I expect to have, at least for the base game).

These are the durations in seconds. 10 runs each, bottom numbers are averages.
One tiny oversight I fixed was moving the Debug.Log call out of the StopwatchScope in order to not have this influence the measurement.
I changed the calls to the MessagePack API to use the normal "MessagePackSerializer" class instead of "LZ4MessagePackSerializer" which I used originally. This way the comparison with Ceras is more fair. The only settings I changed for Ceras were config.DefaultTargets = TargetMember.AllFields; and config.PreserveReferences = false;. In my experience (de-)compression algorithms only take a fraction of the time (de-)serialization algorithms take anyway so adding LZ4 on top should not have much impact for both.
In both cases I created a new instance of the respective serializer just before (de-)serializing. I'll later replace this by a getter property backed by a private field in my custom LevelEditorWindow class in order to reuse the instance.
Can an instance of the CerasSerializer class be serialized by Unity's JsonUtility class btw? This would help when working in the editor since otherwise the backing field would still lose its value whenever play mode is entered or left or scripts are recompiled.

Strike those first 2 sentences, I've definitely been looking in the wrong folder the first time.
See obj instead of bin.
Face -> desk :roll_eyes:

Can an instance of the CerasSerializer class be serialized by Unity's JsonUtility class btw? This would help when working in the editor since otherwise the backing field would still lose its value whenever play mode is entered or left or scripts are recompiled.
I think I understand what you intend to do. You want to minimize the "startup time" as much as possible, so your iteration time is faster (not having to wait as long for your editor thing to load the game/level stuff), right?
Unfortunately there's really not much that can realistically be done there I think.
As for your actual question: There's huge amount of stuff cached inside a CerasSerializer instance, most of which is pretty complex (like static Type-caches, references to dynamically compiled code...) and would be really complicated to preserve/restore.
Anyway, when you enter/leave play mode or recompile scripts many of those things would be invalid anyway and restoring them from a serialization (if that were even doable easily) would not save much time.
For example you obviously can't "serialize" dynamic code, what you'd do is serialize the the construct/"source" from which it was created, and to deserialize it you'd deserialize this sort of description and then compile it again...  All in all, even if someone were to put in the time to somehow make this happen, I highly doubt it would even be worth the effort.
What can be done, however, is further increasing the performance of the serialization/deserialization I think.
But that depends on how exactly your tile object looks like. I assume it's a struct? Can you post the code for it? Also if your goal is performance I'd suggest you try normal nested arrays instead of multi-dimensional arrays as well. At least in the past multidimensional arrays always had quite a few peformance issues. (not 100% sure if that is still the case today).
If your tile object is a struct it could also be a good idea to think about the layout of the data members to ensure there is as little padding as possible (and ensuring that the internal memory layout will not change).

Also let me know if you intend for your game to be cross-platform. Or at least if you intend to, load a save-game on a different platform than it was created. That could be relevant as well (because it would force you to not use any sort of reinterpret formatter, but that affects every serializer, not just Ceras).

I see. Part of the reason for testing with a new instance each time was for this case; if the instance is lost too often it wouldn't make much sense to skip the first measurement, right? It shouldn't be too much of a problem though, since the creating a new instance seems to be fast enough. And since it only happens when the play/edit mode changes or scripts are recompiled it's at least possible to keep the same instance around and (potentially) reuse it multiple times until then.
Interestingly the deserialization time is a bit faster while in play mode where it goes down to ~0.3 seconds compared to the ~0.4 seconds in edit mode.
The Tile class contains these constants and fields:
#region Constants
public const int GraphicsLayersCount = 2;
private static readonly TileTypeComparer tileTypeComparer = new TileTypeComparer();
public static readonly Vector2 Scale = new Vector2(0.5f, 0.5f);
public static readonly Vector2Int PixelSize = new Vector2Int(20, 20);
private static readonly string[] vertexColorNames = { "_Color0", "_Color1", "_Color2", "_Color3" };
private static Mesh mesh;
private static Material typesMaterial;
#endregion
#region Fields
private GraphicsLayer[] graphicsLayers = new GraphicsLayer[GraphicsLayersCount];
private Color color = Color.white;
private Color[] vertexColors = new Color[4];
private HashSet<Type> types = new HashSet<Type>(tileTypeComparer);
#endregion

The TileTypeComparer class derives from IEqualityComparer and is used to prevent boxing when using the Type enum which is used to indicate which Tiles are walls, platforms, water, etc.
The GraphicsLayer struct only contains these fields:
private int tilesetIndex;
private Vector2Int tilesetCoordinate;
This way each Tile contains 2 GraphicsLayers (background and foreground sprites).
It also has a color and 0 to a few Types.
Instead of the 2D Tile array I've tried both using a 1D array with a wrapper property and nested arrays before but both were a bit cumbersome, e.g. when rezising the level.
I'll probably be able to remove the vertexColors array by putting it into the Level class itself. It's used to hold precalculated values from averaged colors of surrounding Tiles to a give a nice smooth look by interpolating these like vertex colors in a custom shader. This is also part of the reason why I'm using this custom Tile class instead of the one in Unity's Tilemap system.
Basically I'm porting an existing game which was written in C++ to Unity by rebuilding everything myself. I mainly do it for fun and learning in order to improve my coding and pick up a few new things along the way.
I've thought about cross-platform before. While the original game was only available for pc it would certainly be rad to build an Android version to be able to play it anywhere. Although then I'd either need to add a custom touch input scheme (difficult due to the amount of buttons needed) or get myself a controller which can be connected to my smartphone :P
Btw: I noticed that the field into which the Tiles get serialized defaults to a size of 33554432 bytes. While I understand the reason for this normally is to prevent repeated allocations/GC runs, in case of my Level class the "serializedTiles" byte array needs to get serialized again using Unity's JsonUtility class where a smaller size would result in better performance.
Would it be possible to instead use the List approach of doubling the size of the array when needed? Although I'm not sure how well this would integrate with compression when a larger array would be needed for serialization beforehand anyway. Could the CerasSerializer instance maybe reserve a byte array as buffer internally for this and after compression copy the result to a second buffer (the one passed by ref in the Serialize method)?
Or should I just use the overload which returns a byte array instead?

Not sure why you'd serialize to Json again. Can you explain?
For the byte array, you are supposed to use the return value (how many bytes are actually written). For example you'd use that to know how many bytes of the array to write to a binary file. But since you're doing some sort of double serialization with Json as a second step that won't really work.
You will have to copy it into a new buffer that has exactly the right size (this is what the other overload does for you. But it prevents you from reusing the buffer then).
You can also customize Ceras' buffer behaviour.
BTW, if you can change your types set into just a singe bit flag enum, remove all managed types from your tile class, and then convert it to a struct, you will be able to gain some insane speed ups really easily. (like 10-100x, or even more... )
Because then ceras can use its reinterpret formatter on your tile array, which turns the serialization into what is essentially a cpblk call (aka memcpy).

Ah, missed a question:
Yes creating an instance of a ceras serializer is not that expensive, but the first time this new instance encounters a new type will be expensive. That means the first serialize or deserialize call (either one populates the caches for both cases) will be noticeable slower than all subsequent calls.
It might be possible to mitigate that a bit if you know exactly what types to expect.
For example you could use the aot generator to generate a .cs file of the code that would get generated dynamically. So you'd save quite  bit of time there (but every change to the targeted type would require you running the aot generator again as the code would be outdated of course). That can be done on a per type basis even.

It doesn't necessarily have to be json. It's just that the [SerializeField] attribute which is used to automatically serialize the annotated field internally uses the JsonUtility class. Which is horribly slow and very limited in its functionality.
Although it's also possible to additionally use the PreferBinarySerialization attribute on a class derived from ScriptableObject which will result in instances of the class being serialized to a binary format instead of json.
But while this is a little bit faster it's still too slow for my purposes which is why I'm doing it manually using a third-party serializer (MessagePack / Ceras) with much better performance results.
My plan is to have all the data for one Level in a single file. For this the class derives from Unity's ScriptableObject class. The Level class then holds the deserialized but volatile Tiles[,] and the binary serializedTiles[] which is saved using the [SerializeField] attribute which this time is fast enough since there are no more custom classes which need to be handled, just a byte array.
The main performance increase however kicks in in combination with compression. This is simply due to the file sizes which have to be hauled around since Unity deserializes, possibly applies changes, and re-serializes a selected asset every single time the editor is repainted in order to draw the Inspector panel.
Here's a comparison with file sizes from JsonUtility:
Normal Json asset: 17.5MB (way too much)
Normal binary asset: 4.7 MB (still too much)
Zipped Json asset: 195 KB (much better)
Zipped binary asset: 15.4 KB (best)
And from a bit of research I did LZ4 might be up to 10 times faster than GZip at which point it becomes more interesting despite the slightly larger output size.
Regarding turning the Tile class into a struct:
The Tile.Type enum actually used to be a byte flag enum. At one point I changed it in order to be able to make it more extendable so that the LevelEditor and the game could be separated enough to make the LevelEditor reusable for other projects. However, lately I'm thinking of changing this back to how it was. Certain parts of the code simply don't allow for enough separation anymore so they are effectively linked together anyway.

I gave it some more thought during the last few days and I think I might be able to change my Level class to not include the serializedTiles[] data (where it would get serialized again) but instead just save it in an extra file, e.g. as a TextAsset with the *.bytes extension which Unity handles as a special case to provide access via a bytes property.
Or I might use a custom file format, e.g. *.map and handle reading and writing manually depending on access speed and possible further functionality; one of my goals is to create the Level / LevelEditor system in such a way that users could potentially create their own maps and share them easily. Having a Scene file which contains a GameObject with a Level component which holds a reference to a Level.bytes data file might not be an optimal solution though either.
I'll need to do some more research and testing on AssetBundles in the next few days to learn whether/how I can create a new AssetBundle which includes a Scene and the data file and distribute it in a way that it can be downloaded at runtime later. The reason why having a Scene per Level would be nice is that I'd be able to keep the default system to place prefabs (e.g. enemies, items, etc.) and drag them around using the Transform component.
However, I'm a bit sceptical about the resulting file size of the Scene. It might be possible to reduce it drastically by setting up a custom system where only 1 Scene is used at all times and all enemies, items and such are just created at runtime from a serialized version which would be placed in the data file beforehand. This approach would however require a custom editor for prefabs placed in the Scene View to be serialized into the data file as well (possibly using custom classes to hold the data) and a custom editor to edit possible values in the Enemy script which would then not be a MonoBehaviour but instead a normal class...
Obviously the first approach would be easier. Maybe the Scene file will be small enough. After all, Tiles vastly outnumber all other LevelObjects so it might be fine.
Would it be possible for the Ceras (de-)serialize methods to get an overload which takes a file path from which to read/ to which to write? Would it be possible/beneficial to use a Stream as well (either internally or provided explicitly) in order to be able to work with very large files where a normal `ReadAllBytes' would reach its limits?

Would it be possible for the Ceras (de-)serialize methods to get an overload which takes a file path from which to read/ to which to write?
That's be just writing using (var fs = File.OpenRead("level.dat")) { ... } and then read the part you're interested in, right? I'm not sure this is something a serializer should do.
Unless I'm misunderstanding (?)
Would it be possible/beneficial to use a Stream as well (either internally or provided explicitly) in order to be able to work with very large files where a normal ReadAllBytes' would reach its limits?`
Good question, giving a good answer to that will take a bit more context:
Initially it might seem like a good idea because Streams make things easy / easier (depending on what you're doing).
However with Stream you always end up with a call to Read(...) (in the deserialize case) which in the end also just reads into a byte array. (And Write using a byte array in the serialize case).
That's why Ceras (and many other high performance serializers) do intentionally not provide a Stream based API. Because it only adds overhead.
Serializing objects requires a lot of tiny reads/writes, so this would be pretty much the worst case scenario for performance.
To deal with large amounts of data that you can't load into memory all at once for any reason, you can (and should) split out individual segments.
With other serializers you'd manually split the object into parts (I think you already suggested something like that, right?). For example you would split your level data into object definitions (what objects there are, where they are and all their other values/properties) and then maybe your tile set into a separate part...
So you'd have two Serialize calls, one for the tiles, one for all the other objects (and maybe generic level/map information like time of day, skybox name, ...)
For more complicated scenarios Ceras also supports something called IExternalRootObject that allows you to automatically deconstruct an object graph into individual parts.
A perfect example for that would be an entire game database.
I've written an example for that already to fully showcase the feature.
To give you a quick idea: Imagine you have a game that has many Unit definitions (Archer, Skeleton, Mage, Orc, Elv, ...) and many Spells and Items and whatnot.
And all those things refer to each other in various ways. For example a summoning-scroll item would have a reference to the specific Unit that it summons/creates when used.
Now when you want to edit anything you'd normally be forced to load everything (or almost everything) because all those objects have all sorts of references to each other, and you can't really load one without the other...
The idea now is that every "root object" (Unit, Spell, Item) has an ID, and you can serialize/deserialize those individually, without having to turn your direct references to IDs.
For example in the summoning-scroll item you'd keep public Unit SummonedUnit; exactly as it is.
When serializing the item Ceras will write the ID of all the root objects instead of the object itself.
That way every object (every unit, spell, and item) will result in its own little byte-array.
When loading one of those again, you can (if you want to) react to the "OnExternalObject " where Ceras gives you the ID and wants you to provide the object for it. (Or you can just return null for a partial load...)
All of this works recursively as well...
Take a look here:
https://github.com/rikimaru0345/Ceras/blob/master/samples/LiveTesting/Tutorial.cs#L327

I gave it some more thought during the last few days and I think I might be able to change my Level class to not include the serializedTiles[] data (where it would get serialized again) but instead just save it in an extra file, e.g. as a TextAsset with the *.bytes extension which Unity handles as a special case to provide access via a bytes property.
Or I might use a custom file format, e.g. *.map and handle reading and writing manually depending on access speed and possible further functionality; one of my goals is to create the Level / LevelEditor system in such a way that users could potentially create their own maps and share them easily. Having a Scene file which contains a GameObject with a Level component which holds a reference to a Level.bytes data file might not be an optimal solution though either.
I'll need to do some more research and testing on AssetBundles in the next few days to learn whether/how I can create a new AssetBundle which includes a Scene and the data file and distribute it in a way that it can be downloaded at runtime later. The reason why having a Scene per Level would be nice is that I'd be able to keep the default system to place prefabs (e.g. enemies, items, etc.) and drag them around using the Transform component.
However, I'm a bit sceptical about the resulting file size of the Scene. It might be possible to reduce it drastically by setting up a custom system where only 1 Scene is used at all times and all enemies, items and such are just created at runtime from a serialized version which would be placed in the data file beforehand.

I think a custom file format is a good idea.
One thing I've learned when dealing with Unity over the years, is that taking more control over your data is always good. Personally I only use Unitys built-in serialization mechanisms (scenes, scriptable objects, ...) for extremely simple stuff (if at all).
Whenever you want to maintain references to scene objects you'll very quickly run into issues.
In my experience at least it's worth the effort of simply biting the bullet and doing everything manually.
The way you described it here is already the first (and biggest) step: ...setting up a custom system where only 1 Scene is used at all times and all enemies, items and such are just created at runtime from a serialized version which would be placed in the data file....
If you're already willing to do that, then the remaining steps of completely decoupling everything from Unity/Scenes is relatively straightforward.

This approach would however require a custom editor for prefabs placed in the Scene View to be serialized into the data file as well (possibly using custom classes to hold the data) and a custom editor to edit possible values in the Enemy script which would then not be a MonoBehaviour but instead a normal class...
Obviously the first approach would be easier. Maybe the Scene file will be small enough. After all, Tiles vastly outnumber all other LevelObjects so it might be fine.

I get where you're going with this, but it sounds like a huge amount of work.
Wouldn't you have to write a completely custom unity-inspector for almost every object then?
Also making a custom inspector window that allows you to not only change simple stuff like strings and numbers, but also change references to other objects sounds pretty complicated...
But maybe all of this isn't needed.
From what I understand what you're actually looking for is just a way to customize how things are saved (serialized).

Then, instead of saving the scene as usual, you'd have a custom script that collects all objects in the scene, and writes them to a custom serialized format.
You could find each gameobject, from there each monobehaviour (just call GetComponents<Component>() to get literally every component on the gameobject), and from there you'd just call GetType() on each component to get the actual type, and once you have the type you can iterate through all its public fields and properties.
Then saving would be easy.
In its most simple form (just to illustrate my point) you could then just serialize everything as a  Dictionary<string, object>.
Maybe like this:
class MySerializedGameObject
{
   public string Name;
   public Matrix4x4 Transform;
   public List<MySerializedComponent> Components;
}

class MySerializedComponent
{
  public string ComponentTypeName;
  public Dictionary<string, object> FieldAndPropertyValues;
}

This will work for simple stuff, where components only contain simple types.
But extending it so references from one GameObject to another are correctly maintained shouldn't be too hard either. (just add some sort of class MyGameObjectId : MonoBehaviour ... thing to every gameobject so you can refer to gameobjects...).
Actually... now that I think about it... I think Ceras can do most of the more complicated work for you.
I haven't fully thought this through yet, but if someone were to write a custom IFormatter<> for UnityEngine.GameObject and UnityEngine.Component then all of this could probably be solved really easily and much quicker than I initially thought.
Ceras would handle pretty much everything; mapping GameObjects and components to IDs and back, and also taking care of any other data types (even the ones that neither Unitys internal binary serializer, YAML/Json serializer supports).
You'd be completely side-stepping Unitys serialization.
You wouldn't have to write any custom editor tools, everything stays GameObject+MonoBehaviour(Component).
Only saving / loading would be custom.
And you could even mix things. Simply excluding certain GameObjects from getting serialized in your custom serialization would be very easy. You could create a scene/level with terrain and trees and whatever, and then have all the actual "entities" on the map (dynamic stuff like monsters, items, ...) serialized into the custom format...
Anyway, I'm not entirely sure if all of this would really work as I imagine it, but I think it'd be relatively little work to find out / make a prototype. A prototype will definitely reveal any edge-cases or things I didn't consider. Maybe I'll even try it myself when I have some time :P

one of my goals is to create the Level / LevelEditor system in such a way that users could potentially create their own maps and share them easily. Having a Scene file which contains a GameObject with a Level component which holds a reference to a Level.bytes data file might not be an optimal solution though either

At runtime? Like, having an ingame-level editor?
Creating new assets like an 'AssetBundle' file at runtime is impossible anyway (so unless you roll your own thing, you'd definitely not be able to have a runtime-editor anyway). Same thing goes for 'Scene', you can create a new empty scene, but you can't save scenes at runtime.
The idea I outlined above would make that possible though.
Or maybe I misunderstood and your plan is to have people download Unity and install some sort of editor-addons you made, so they can make levels for your game (?).

That'd be just writing using (var fs = File.OpenRead("level.dat")) { ... } and then read the part you're interested in, right? I'm not sure this is something a serializer should do.
Yeah, you're probably right. Especially if a Stream API overload also doesn't provide any benefit.
It seems saving the serialized data to a file comes at the trade-off of not being able to use a buffer then, at least for infrequent events such as loading the Tile data of a Level.

I've had a look at the example to split large reference structures. It's pretty cool tech but probably not needed for my project ;)
Instead I did a fair amount of refactoring in the last few days to optimize my types as you suggested. Hence the delayed reply, sorry about that. I wanted to see it in a working state though first in order to be able to make a new performance test:

So saving now is 10+ times faster while loading is 23+ times faster which is pretty nice. And even with the usual editor overhead. At this point I think I might have to set an artificial minimum time for the loading screen :D
Something I'm looking out for though is the size of the struct. I've read that the recommended size is 16 bytes or only slightly more since at some point copying a struct would get more expensive than using a class object. Having ultra fast loading/saving speeds at the cost of reduced performance during runtime when objects get passed around all the time would not be ideal...
Currently the Tile struct holds a GraphicsLayerCollection struct (in order to not use a List or Array since those would introduce references into the struct, leading to it being placed in the heap instead of the stack - or am I mistaken?) which uses an indexer property to grant access to the 2 GraphicsLayer structs it contains which in turn have an int for the Material index and a Vector2Int for the coordinate.
The Tile also has a Color and a Tile.Type flags enum (instead of a Hashset now).
This results in 2 * (1 + 2) * 4[number of bytes for an int] + 4 + 1 = 29 bytes which might be a bit much, no? From my understanding passing Tiles via ref keyword would mean that they wouldn't have to be copied but I'm not sure yet whether this will be possible everywhere in my code due to the way I access them at certain points.
One more thing I did was changing the 2D Tile[,] array to a jagged Tile[][] array which should also help with performance. Since changing Tile from a class to a struct broke my Size() extension method which worked on object[,] arrays I figured I might as well. Still, not having this extension method is a bit of a PITA since I've had to replace quite a few lines by writing new Vector2Int(Tiles.Length, Tiles[0].Length) manually instead of just calling this:
public static Vector2Int Size(this object[,] array)
{
	return new Vector2Int(array.GetLength(0), array.GetLength(1));
}

I also had a look at the resulting file size: Using a test Level with 650256 Tiles results in 6.98MB whereas 65025629 would only result in 4.6MB. I guess the diffence comes from additional type data?
Zipping the file results in a size of 255KB which is larger than the 15.4KB from the zipped binary file I was testing before. But then again that particular map was just 700400 empty Tiles whereas this 650*256 map actually contains a real level with varying values in the GraphicsLayers. I'm guesing this results in less-compressable data because certain values/blocks aren't repeated endlessly anymore?

I think a custom file format is a good idea.
I tested saving as *.bytes and *.map which both resulted in the same file size. So I'll probably test whether using the bytes property Unity provides for the *.bytes file is better or worse than loading the *map file myself, performance-wise.

Whenever you want to maintain references to scene objects you'll very quickly run into issues.
Yep, I noticed this when I tried to turn my Level class from a ScriptableObject into a MonoBehaviour in order to see whether I could just save it in a scene which I would then load additively to the main scene which contains the game manager class. However, this resulted lost references and the editor not even allowing me to reassign them (at least without using EditorSceneManager.preventCrossSceneReferences and some lazy initialization properties in order to find the references at runtime...). It puzzles me a bit that Unity doesn't allow extending the Scene class in order to save additional custom data in a "global" sense. Instead you always have to place such data in a MonoBehaviour component on a GameObject for which you then have to search and hope it exists. Which doesnt exactly look like type-safety...

The way you described it here is already the first (and biggest) step: [...]
If you're already willing to do that, then the remaining steps of completely decoupling everything from Unity/Scenes is relatively straightforward.
To be honest, I'd like to try and avoid piling extra work on top of everything that's already still ahead of me (a custom sprite animation system, a custom physics system, a player controller, porting all enemy behaviors, ...) :P
While my goal is more to write good code instead of getting done faster (so I don't need to search a new hobby again soon ;) ) I'd like to see some progress at some point.

How about something like this then: [...]
Maybe like this: [...]
This is pretty much what I had in mind, yes.
There are some additional values to save for GameObjects (parent-reference, name, tag, layer) but apart from that it seems good.

Actually... now that I think about it... I think Ceras can do most of the more complicated work for you.
I haven't fully thought this through yet, but if someone were to write a custom IFormatter<> for UnityEngine.GameObject and UnityEngine.Component then all of this could probably be solved really easily and much quicker than I initially thought.
Ceras would handle pretty much everything; mapping GameObjects and components to IDs and back, and also taking care of any other data types (even the ones that neither Unitys internal binary serializer, YAML/Json serializer supports).
You'd be completely side-stepping Unitys serialization.
You wouldn't have to write any custom editor tools, everything stays GameObject+MonoBehaviour(Component).
Only saving / loading would be custom.

This sounds absolute amazing to the point where you should contact the Unity devs to hire you to throw out their old serializer and replace it with Ceras :D
Apparently, a lot of stuff (especially editor-side) was built on top of the old serializer over the years and either there's noone left who knows how to improve it or they don't dare touch it anymore because everything is held together by duct tape and prayers :P
One thing I'm not sure about however is deserialization; while it's possible to call new GameObject(), calling AddComponent<T>() would then just add a new instance with default values. Would it be possible to overwrite those values even if they're not accessible (private/protected)?

At runtime? Like, having an ingame-level editor?
Nah, nothing that fancy. Building the EditorWindow for my LevelEditor is enough work without having to think about getting actual graphics for an ingame UI and creating an even more simplified layout than I already have in order for it to be usable with any ingame controls.
Or maybe I misunderstood and your plan is to have people download Unity and install some sort of editor-addons you made, so they can make levels for your game (?).
This would work better for me. I guess by "create their own maps and share them easily" I meant relatively easy ;)
But I think it should be possible to set up a system to check an online resource (e.g. a file on github) to get a list of veryfied maps/map packs (name, version, download link, hash code for security check, etc.).
This way map creators could upload their map and create a pull request on my master file which I could then check and approve. Players would then see an ingame menu with a button to check for custom maps which downloads the master file and displays the names of the available maps with a download button next to them (or a play button if it's already downloaded).
Maybe with an option to load a map from a local folder for people without local internet connection and who are willing to take the risk of installing maps with penis drawings as level design :P

Yeah, you're probably right. Especially if a Stream API overload also doesn't provide any benefit.
It seems saving the serialized data to a file comes at the trade-off of not being able to use a buffer then, at least for infrequent events such as loading the Tile data of a Level.

If it ever becomes a problem that can be fixed relatively easily as well.
Btw, I was experimenting with all sorts of compression stuff. Like how various compression libraries (like zlib, or LZ4) could be integrated with Ceras.
It turns out that there isn't anything that could be done that would make a big difference.
After you're done with the serialization, you'd just do something like LZ4Codec.Encode(source, target);, or use a LZ4CompressionStream and then write to that if you want to (doesn't make much of a difference either way)...
It's pretty much the same thing as with writing to files / reading from them. Integrating it with the serializer doesn't offer any potential benefits.

So saving now is 10+ times faster while loading is 23+ times faster which is pretty nice. And even with the usual editor overhead. At this point I think I might have to set an artificial minimum time for the loading screen :D

Looks great! Those are some insane improvements!

Something I'm looking out for though is the size of the struct. I've read that the recommended size is 16 bytes or only slightly more since at some point copying a struct would get more expensive than using a class object. Having ultra fast loading/saving speeds at the cost of reduced performance during runtime when objects get passed around all the time would not be ideal...

My advice: don't waste any time thinking about the struct size at all. That microsoft article (pretty sure that's where that idea is from) is pretty old and, by now, irrelevant. Just pass structs by reference and you'll be fine.

Currently the Tile struct holds a GraphicsLayerCollection struct (in order to not use a List or Array since those would introduce references into the struct, leading to it being placed in the heap instead of the stack - or am I mistaken?)

Yeah, basically the performance increase comes from the fact that Ceras detects that the struct is "blittable" (a word which has many conflicting definitions, there's not really a universally agreed upon definition for it) and that allows it to use its ReinterpretFormatter.
I could go into great detail how all this works, but to make it short, there's only one thing to keep in mind: The target type must be a struct, and the struct must not contain any non-blittable types.
Examples of non-blittable types are: string, any sort of array T[], anything that is defined as class, ... (the array part might be a bit confusing the content can be reinterpreted as long as the element-type is blittable, but the array object itself is not blittable. That's why having any sort of array inside a struct makes the whole struct non-blittable. If something is unclear don't hesitate to ask.)

This results in 2 * (1 + 2) * 4[number of bytes for an int] + 4 + 1 = 29 bytes which might be a bit much, no? From my understanding passing Tiles via ref keyword would mean that they wouldn't have to be copied but I'm not sure yet whether this will be possible everywhere in my code due to the way I access them at certain points.

The size doesn't matter at all, this is not the place to optimize. Just pass them by ref.
Alternatively you can pass the tile array + the x,y coordinates as well (which is like passing by ref, just "manually" in a way :P)
due to the way I access them at certain points.
We'd need a concrete/specific example here. It should always be possible.

One more thing I did was changing the 2D Tile[,] array to a jagged Tile[][] array which should also help with performance.

I tried to make a reinterpret formatter for multidimensional arrays work. But it turned out way too brittle. But I thought of some new ideas that could make it work. However implementing those will probably take quite a bit of time (that I don't really have :P). Also since you've already refactored to jagged arrays it's not really needed anymore anyway.

Since changing Tile from a class to a struct broke my Size() extension method which worked on object[,] arrays I figured I might as well.

Wouldn't the following work?
public static Vector2Int Size(this Tile[][] array)
{
	return new Vector2Int(array.Length, array[0].Length);
}

I also had a look at the resulting file size: Using a test Level with 650x256 Tiles results in 6.98MB whereas 650x256 x29 would only result in 4.6MB. I guess the diffence comes from additional type data?

I have no idea. I very much doubt that any type data would cause this though. Ceras is very efficient with its type encoding system. What does the "x29" mean?

I'm guesing this results in less-compressable data because certain values/blocks aren't repeated endlessly anymore?

Depends on the compression algorithm. But yeah, probably.

I tested saving as *.bytes and *.map which both resulted in the same file size.

The file extension doesn't do anything. I probably should have said "serialized format" instead of "file format". By using Ceras (or any serializer) you're already essentially using a custom format.

To be honest, I'd like to try and avoid piling extra work on top of everything that's already still ahead of me (a custom sprite animation system, a custom physics system with a custom collision system, a player controller, porting all enemy behaviors, ...) :P
While my goal is more to write good code instead of getting done faster (so I don't need to search for a new hobby again soon ;)) I'd like to see some progress at some point.

Haha yeah I can totally understand that.

This sounds absolutely amazing to the point where you should contact the Unity devs to hire you to throw out their old serializer and replace it with Ceras :D
Apparently, a lot of stuff (especially editor-side) was built on top of the old serializer over the years and either there's noone left who knows how to improve it or they don't dare touch it anymore because everything is held together by duct tape and prayers :P

I don't think it's that easy (for Unity at least), but I think you're right. It could definitely be something to replace GameObject/Scene serialization. But then again Unity is pushing their ECS stuff pretty heavily. One of its advantages is the ability to do pretty much what my ReinterpretFormatter does. So serialization/deserialization/save/load will be much faster for ECS stuff anyway.

One thing I'm not sure about however is deserialization; while it's possible to call new GameObject(), calling AddComponent() would then just add a new instance with default values. Would it be possible to overwrite those values even if they're not accessible (private/protected)?

Yeah, Ceras can overwrite members/values of existing instances, including private members, and even readonly fields! :D

Btw, I was experimenting with all sorts of compression stuff. Like how various compression libraries (like zlib, or LZ4) could be integrated with Ceras.
It turns out that there isn't anything that could be done that would make a big difference.
After you're done with the serialization, you'd just do something like LZ4Codec.Encode(source, target);, or use a LZ4CompressionStream and then write to that if you want to (doesn't make much of a difference either way)...
It's pretty much the same thing as with writing to files / reading from them. Integrating it with the serializer doesn't offer any potential benefits.

You mean in terms of further performance improvements when passing the data and using/not using Streams, right?
Not in the sense that applying LZ4 does not result in a smaller byte array/file size? The latter would surprise me, seeing how e.g. a 7152 KB file containing a serialized Level can be packed into a 457 KB zip file (in the Windows Explorer).

Wouldn't the following work?
public static Vector2Int Size(this Tile[][] array)
{
return new Vector2Int(array.Length, array[0].Length);
}

Yep, this works. It doesn't have the flexibility to be used on any object anymore, but oh well. I went through my code and changed the lines back where I used the explicit calculations each time. Although I'm thinking of wrapping the jagged array and these extention methods into a TileGrid class with a Vector2Int indexer property. This would allow me to change the extention methods to getter properties instead as well.

I have no idea. I very much doubt that any type data would cause this though. Ceras is very efficient with its type encoding system. What does the "x29" mean?

That's the 29 bytes per Tile.

The file extension doesn't do anything. I probably should have said "serialized format" instead of "file format". By using Ceras (or any serializer) you're already essentially using a custom format.

Yeah, this was just to test whether Unity would do anything with it after it recognized that there was a new file. One thing I still need to test is AssetBundles which can have no compression, LZMA or LZ4 compression applied when building them. I'm still getting a few compiler errors though which I'll need to fix in order to build one.

I don't think it's that easy (for Unity at least), but I think you're right. It could definitely be something to replace GameObject/Scene serialization. But then again Unity is pushing their ECS stuff pretty heavily. One of its advantages is the ability to do pretty much what my ReinterpretFormatter does. So serialization/deserialization/save/load will be much faster for ECS stuff anyway.

I'd need to have another look at ECS and what its current status is. However, from what I remember when I had a look at it in the past it didn't seem trivial to use (or at least not as trivial as Ceras) to the point where wanting to use it would require a serious justification (i.e. performance increase) in order to make it worth the hassle.

You mean in terms of further performance improvements when passing the data and using/not using Streams, right?
Not in the sense that applying LZ4 does not result in a smaller byte array/file size? The latter would surprise me, seeing how e.g. a 7152 KB file containing a serialized Level can be packed into a 457 KB zip file (in the Windows Explorer).

Yeah exactly :D Sorry for the confusion! I was just talking about how I'm having a hard time "combining" Ceras and LZ4 more tightly.
It seems to me that the current approach (where you'd just call Serialize() and then pass the data to LZ4) is already as optimal as it gets. Like there's no overhead there that could fixed by a more "in-depth" integration between the two libraries.

I'd need to have another look at ECS and what its current status is. However, from what I remember when I had a look at it in the past it didn't seem trivial to use (or at least not as trivial as Ceras) to the point where wanting to use it would require a serious justification (i.e. performance increase) in order to make it worth the hassle.

Oh yes indeed, I completely agree!
It's a lot more complicated than the classical way of programming for sure.
Maybe it will get a little bit better in the future, but I don't expect any huge improvements in terms of usability / comfort.

It seems to me that the current approach (where you'd just call Serialize() and then pass the data to LZ4) is already as optimal as it gets. Like there's no overhead there that could fixed by a more "in-depth" integration between the two libraries.

Would you still build these compression libraries into Ceras then, for comfort of use? Or would it make more sense to maybe just have some links here on github to lead to other repositories where the respective compression libraries can be downloaded, in order to keep Ceras a bit smaller? I'm not sure how much of a size difference it would make. And if I understand this correctly Unity will strip away unused libraries when building with IL2CPP?

Ok so help me with one thing here please. I'm having a hard time coming up with any ideas of what to do with any compression libraries even.
Like, lets say I'd create a new nuget package named Ceras.LZ4CompressionAddon.
What would that package even do?? :laughing: :sweat_smile:
I think an example could help in showing what I mean. Lets say we are serializing some data and then saving it to a file. But now we want to also add LZ4 compression ontop of that, right?
But then doing that would be just:

Before

byte[] buffer = null;
int dataLength = ceras.Serialize(myLevel, ref buffer);

using(var file = File.Create("someData.bin"))
    file.Write(buffer, 0, dataLength);

After

byte[] buffer = null;
int dataLength = ceras.Serialize(myLevel, ref buffer);

using(var file = File.Create("someData.bin"))
    using(var lz4Stream = LZ4Stream.Encode(file))
       file.Write(lz4Stream , 0, dataLength);

The difference in the code is literally just a single line. And LZ4Stream would be available after two clicks by just installing K4os.Compression.LZ4.Streams from NuGet.
So now what I don't get is this: What could I possibly write in my little hypothetical Ceras.LZ4CompressionAddon to make that any easier? Or more "integrated"?
Would I just make a wrapper for LZ4Stream? But what for? My addon library would have a dependency on K4os.Compression.LZ4.Streams anyway, and most likely my wrapper wouldn't be as feature complete as using the original thing. So what's the point :joy:
Is there anything obvious I'm missing?

Hmm, I guess if this can be done in one line there's not much point in including it.
It seems in neuecc's MessagePack serializer which I was using before this was a bit more work which is why he a added an extra class?
I should have some free time during the next few days so I could try adding the LZ4 library to my project. Would it be enough to just add these C# classes or do I need to download and build the project via NuGet?

It seems in neuecc's MessagePack serializer which I was using before this was a bit more work which is why he a added an extra class?

I think explaining the real reason why that is neccesary for MessagePack-CSharp requires a bit of an explanation:
"MessagePack" is actually a format definition while and "MessagePack-CSharp" is an implementation of that format.
That means the MessagePack-CSharp library has to - in order to stay compatible to the standard - wrap the compressed data into a "custom data" description. So other libraries trying to read that data can at least say "ok, this is something special that's not defined in the spec and I can't read it". Like for example some Python application can then just skip the data (the wrapper won't really tell the reading application how to decompress or read it, but at least it won't crash trying to read).
Ceras on the other hand does not define a public format, resulting in a different set of advantages and disadvantages. Most importantly:

Disadvantage: Its only implemented for .NET (C#, F#, VB, Boo, ...). It is not quite so easy for any developer to come along and just make a "Ceras.js" or "Ceras for python" library.

Advantage: because there is no spec/definition that anyone expects me to adhere to, I can do much more optimizations and add features much more quickly.

Same: Ceras is cross-platform. It runs everywhere .NET or IL2CPP are available (Windows, Linux, Android, iOS, Ps4, Switch, ...). And data serialized using Ceras on any platform can be read on any other platform (just like with any other format like MessagePack, JSON, XML, HTML, ...)

I should have some free time during the next few days so I could try adding the LZ4 library to my project. Would it be enough to just add these C# classes or ...

You can't copy just those classes you linked. They just define the stream API, but there's much more to LZ4 than that.

...do I need to download and build the project via NuGet?

You could either download K4os.Compression.LZ4 as source code and then compile it yourself; if you want to, but there's not really any reason to do that.
Or you can just install the already compiled (dll-)version by using NuGet (what almost everyone does). In which case you'd do no building/compiling of the LZ4 library on your own. You'd just add the using K4os....  thing to the top of your cs file and then use the library like in my example.
I hope that helps :smile:  :+1:

That means the MessagePack-CSharp library has to - in order to stay compatible to the standard - mark the compressed data as being an "extension" type (extension of the msgpack format). So other libraries know that the data that follows is not something defined in the standard, and they need to have special handling to understand it (or just error out, which is what actually happens in most cases).

Okay, that would explain it.
Regarding the K4os library:
I downloaded the normal and the Stream package, unzipped them and copied the dll from the lib folder into the Plugins folder in my project.
But either I was doing something wrong again or they are incompatible with Unity; just having the using K4os... line in a script results in Unity throwing this error message:

Assembly 'Assets/Plugins/K4os.Compression.LZ4.dll' will not be loaded due to errors:
Unable to resolve reference 'System.Memory'. Is the assembly missing or incompatible with the current platform?
Reference validation can be disabled in the Plugin Inspector.

Also the lines from the "After" code you posted above gave me some errors, e.g. the File.Write method not being able to accept an LZ4Stream as first parameter... And also the buffer doesn't get used there anymore?
However, I've found a different solution:
Since the data gets saved to a file now I don't need to worry about performance as much anymore because the majority of the data (~7MB for the Tiles of the current Level) doesn't need to get serialized again by Unity every time the Inspector gets repainted but instead only at single events (e.g. when pressing a "Save" button in the LevelEditor UI. Same for loading.
I also did some refactoring in order to get my project to build so I could test AssetBundles a bit more and it turns out that I should be able to use the compression formats they provide (LZ4 or LZMA).
Basically I turned the Level class from a ScriptableObject into a MonoBehaviour and added it to an empty GameObject in my Scene. The Level also contains a TextAsset variable now which holds a reference to the file which is created when saving the Tiles. This way I'm able to build everything relevant (the Scene, in order to be able to manually place enemy prefabs etc; the Level, to act as a conroller; and the TextAsset for the Tile data) into a single AssetBundle. (At the moment some referenced Materials and Textures also get pulled into the same AssetBundle automatically but I'll do some more research to be able to separate them into their own AssetBundle and find a dependency structure which works best for me.)
I guess this should solve it then and we could close this issue? If someone has any more questions in the future they could just reopen it.
And thank you for your help with the optimizations again! I'll look into a few more things (e.g. KnownTypes) and if anything else comes up I'll just open another issue ;)

But either I was doing something wrong again or they are incompatible with Unity; just having the using K4os... line in a script results in Unity throwing this error message:

That's probably because System.Memory depends on something else as well (and that dependency is assumed to be part of the runtime, but when used in Unity it's not there).
I'd suggest just opening an issue about that in the github page of the K4os.Compression.LZ4 lib.

Also the lines from the "After" code you posted above gave me some errors, e.g. the File.Write method not being able to accept an LZ4Stream as first parameter... And also the buffer doesn't get used there anymore?

Yeah that was just an example I typed out directly (not in visual studio :laughing: ).
I think the last line should just be

lz4Stream.Write(buffer, 0, dataLength);

I guess this should solve it then and we could close this issue? If someone has any more questions in the future they could just reopen it.
And thank you for your help with the optimizations again! I'll look into a few more things (e.g. KnownTypes) and if anything else comes up I'll just open another issue ;)

Sure, close it :)
No problem, if you have any questions about KnownTypes make sure to take a look at the new wiki page about type encoding here: https://github.com/rikimaru0345/Ceras/wiki/Type-Encoding
Maybe it's helpful for what you're trying to do.


Halting while waiting for API server

Hey, first of all, this repo is awesome. Thank you so much for sharing it!
I've been reading through everything and I think I have a handle on the issue I'm seeing, but I'm still new to K8s and I'm not sure where to begin trying to solve it. So I'm hoping for some support here, more than calling this a "bug".
Basically, I'm seeing the terraform process halt during the initialize step when it's doing the remote curl http://<IP_ADDRESS>:8080/ – upon inspection, I don't see the kube API server running on any of the nodes, so it makes sense that it's stuck there. I see that the intention is to have an HA-setup of the kube API service, so I gather we're doing things differently than might be normal, here. I also see that there is a kube-apiserver.yaml k8s manifest, which leads me to believe that we're trying to have k8s run its own API service (as opposed to running the API service directly with systemd). The confusion I'm having now is about how that's supposed to work. I see in the kubelet logs   that it's also freaking out about not being able to talk to the API, but as far as I can tell, it's the kubelet which would be the only process aware of the k8s manifests and which should ultimately schedule the API service.
Thanks in advance for being patient with my noobiness. :smile:

hey ryan! you are welcome :) i figured the community could benefit from a terraform reference implementation...
could you share a few lines from:

the console output around where the problem is happening ?
journalctl on <IP_ADDRESS> where you see the freak out about not being able to talk to the apiserver

it seems the trendy thing to do these days is let kubelet run apiserver, proxy, controller-manager et al as pods. the --config switch to kubelet indicates a directory which is scanned for changes every 20s or so (not sure the exact timing). see modules/etcd/cloud-config.tf:
    - name: kubelet.service
      command: start
      content: |
        [Unit]
        After=docker.socket
        ConditionFileIsExecutable=/opt/bin/kubectl
        Requires=docker.socket
        [Service]
        ExecStart=/opt/bin/kubelet \
          --allow-privileged=true \
          --api-servers=http://<IP_ADDRESS>:8080 \
          --cluster-dns=<IP_ADDRESS> \
          --cluster-domain=cluster.local \
          --config=/etc/kubernetes/manifests \
          --register-node=false
        Restart=always
        RestartSec=5
        [Install]
        WantedBy=multi-user.target

if you do docker ps on <IP_ADDRESS> you should see the containers that kubelet launched there (btw kubelet will keep these running should they die):
$ make ssh
$ ssh <IP_ADDRESS>
$ docker ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS               NAMES
ca67ec671116        gcr.io/google_containers/hyperkube:v1.2.0   "/hyperkube scheduler"   6 minutes ago       Up 6 minutes                            k8s_kube-scheduler.7b10907e_kube-scheduler-ip-10-0-0-10.us-west-1.compute.internal_kube-system_1adf66daf2f3ee9cce476b5b97a8b58c_114ac564
a4eee538f5c3        gcr.io/google_containers/hyperkube:v1.2.0   "/hyperkube controlle"   6 minutes ago       Up 6 minutes                            k8s_kube-controller-manager.89f70b8f_kube-controller-manager-ip-10-0-0-10.us-west-1.compute.internal_kube-system_c79711b2d60ce63ccce896a8111c8ba3_f4299389
d07481055ccd        gcr.io/google_containers/hyperkube:v1.2.0   "/hyperkube proxy --m"   6 minutes ago       Up 6 minutes                            k8s_kube-proxy.51156269_kube-proxy-ip-10-0-0-10.us-west-1.compute.internal_kube-system_569f2def3d8e2df8825f616eba237006_319e3343
dd7c1bf2370e        gcr.io/google_containers/hyperkube:v1.2.0   "/hyperkube apiserver"   6 minutes ago       Up 6 minutes                            k8s_kube-apiserver.57925c18_kube-apiserver-ip-10-0-0-10.us-west-1.compute.internal_kube-system_adbb9b34587af41c37b3c16ab2b881a6_225abce7
f611f81b68de        gcr.io/google_containers/pause:2.0          "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD.6059dfa2_kube-scheduler-ip-10-0-0-10.us-west-1.compute.internal_kube-system_1adf66daf2f3ee9cce476b5b97a8b58c_2de18374
3369a7afc846        gcr.io/google_containers/pause:2.0          "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD.6059dfa2_kube-controller-manager-ip-10-0-0-10.us-west-1.compute.internal_kube-system_c79711b2d60ce63ccce896a8111c8ba3_3f676c9c
009ca4840acb        gcr.io/google_containers/podmaster:1.1      "/podmaster --etcd-se"   6 minutes ago       Up 6 minutes                            k8s_controller-manager-elector.433b6cc4_kube-podmaster-ip-10-0-0-10.us-west-1.compute.internal_kube-system_66f8cf4ddf87288d1bbbb588a4bc6077_d9878022
250152b90047        gcr.io/google_containers/podmaster:1.1      "/podmaster --etcd-se"   6 minutes ago       Up 6 minutes                            k8s_scheduler-elector.c6c6198_kube-podmaster-ip-10-0-0-10.us-west-1.compute.internal_kube-system_66f8cf4ddf87288d1bbbb588a4bc6077_d2c2f1dc
a627370bf6a1        gcr.io/google_containers/pause:2.0          "/pause"                 7 minutes ago       Up 7 minutes                            k8s_POD.6059dfa2_kube-proxy-ip-10-0-0-10.us-west-1.compute.internal_kube-system_569f2def3d8e2df8825f616eba237006_896345a6
653a751edddf        gcr.io/google_containers/pause:2.0          "/pause"                 7 minutes ago       Up 7 minutes                            k8s_POD.6059dfa2_kube-apiserver-ip-10-0-0-10.us-west-1.compute.internal_kube-system_adbb9b34587af41c37b3c16ab2b881a6_c7fc5770
fd4512b3b604        gcr.io/google_containers/pause:2.0          "/pause"                 7 minutes ago       Up 7 minutes                            k8s_POD.6059dfa2_kube-podmaster-ip-10-0-0-10.us-west-1.compute.internal_kube-system_66f8cf4ddf87288d1bbbb588a4bc6077_70867fa5

that delay you are seeing is normal - i have seen it take a minute or so for services in /etc/kubernetes/manifest to start. docker is pulling the containers etc.
i take it you have waited for more than a few minutes though.
lmk if this helps and send across those logs and console snips.
shouldn't be too hard to sort this out.

oh also... does systemctl status on <IP_ADDRESS> show any issues ?

Thanks for your help! Here's a gist with the logs you wanted: https://gist.github.com/rtlong/2ddd2171c230078c368a
I can't give you the TF output as I've lost it in my shell now, but it was definitely showing the null_resource.initialize lines where it described the connection details, followed by (presumable a parallel worker produced this) a block from another resource, then the solitary line, again from null_resource.initialize which said merely Connecting! and that's the last thing I see before I killed it after 15 minutes or so. (Also possibly of note: I had to ^C twice to break out of that. The first time TF shows it's trying to shutdown cleanly, but it never happens. I assume it's waiting for all active provisioners to finish, so that would make sense.)

np :)
hey remove this (or edit them) from the cloud-config.tf files in the worker and etcd modules:
        - name: awslogs.conf
          content: |
            [Service]
            Environment="DOCKER_OPTS=--log-driver=awslogs --log-opt awslogs-region=us-west-1 --log-opt awslogs-group=${ log-group }"

i have hard coded the region - DOH!!! definitely a bug !
anyway remove them and it should work but won't log to cloudwatch. if you are using a different region try editing to that.
here are the offending log entries:
Mar 23 22:45:59 ip-10-0-0-10.ec2.internal kubelet[1132]: E0323 22:45:59.416603    1132 manager.go:1864] Failed to create pod infra container: API error (500): Cannot start container fac40eae07a74838b27fcd0246a69315df15637c5f79205254780850712d2658: Failed to initialize logging driver: ResourceNotFoundException: The specified log group does not exist.
Mar 23 22:45:59 ip-10-0-0-10.ec2.internal kubelet[1132]: status code: 400, request id: []
Mar 23 22:45:59 ip-10-0-0-10.ec2.internal kubelet[1132]: ; Skipping pod "kube-proxy-ip-10-0-0-10.ec2.internal_kube-system"
Mar 23 22:45:59 ip-10-0-0-10.ec2.internal kubelet[1132]: E0323 22:45:59.416678    1132 pod_workers.go:112] Error syncing pod e153596e1cb9c8b7548d10d172f3beea, skipping: API error (500): Cannot start container fac40eae07a74838b27fcd0246a69315df15637c5f79205254780850712d2658: Failed to initialize logging driver: ResourceNotFoundException: The specified log group does not exist.

can you check that terraform created a log group for the intended region ?
also delete your gist in case there is anything sensitive in there.
i'll fix this properly tomorrow.
lmk how it goes

That did the trick! Thanks. I'm so glad I reached out. I usually like to spend more time trying to understand something like this before asking for help, but that would have taken me a while to find, I'd guess. I was way off on the wrong track.

Np anytime.

FYI there's also a hard-coded region in init-cfssl. Now that I've fixed that as well, I was able to get this to complete, and now have a happy kubectl telling me my k8s cluster is up and running. Thanks again for sharing this with the world. I'm trying to get going with the latest pre-release of Deis, so needed to get a k8s cluster going. This has been a helpful jump start, as I know I'm going to us TF/CoreOS on AWS anyway.

Yeah damn - I realized that too just an hour ago :(
Glad this helped !!

On Mar 24, 2016, at 5:13 PM, Ryan Taylor Long<EMAIL_ADDRESS>wrote:
FYI there's also a hard-coded region in init-cfssl. Now that I've fixed that as well, I was able to get this to complete, and now have a happy kubectl telling me my k8s cluster is up and running. Thanks again for sharing this with the world. I'm trying to get going with the latest pre-release of Deis, so needed to get a k8s cluster going. This has been a helpful jump start, as I know I'm going to us TF/CoreOS on AWS anyway.
—
You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub


[SPARK-25141][SQL][TEST] Modify tests for higher-order functions to check bind method.

What changes were proposed in this pull request?
We should also check HigherOrderFunction.bind method passes expected parameters.
This pr modifies tests for higher-order functions to check bind method.
How was this patch tested?
Modified tests.

cc @mn-mikke @mgaido91

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2268/
Test PASSed.

Merged build finished. Test PASSed.

Test build #94883 has started for PR 22131 at commit 8154bf4.

I am not sure about this. I think that the bind method itself is tested with the end-to-end tests which have been added. This complicates quite this part which is meant only to test the execution of the functions. I don't see much value added by this tests, but I may miss something. Can you enlighten me @ueshin ? Thanks.

Actually one of my motivations is to prevent a mistake like #22126. To create a test, we needed to do the same thing in bind. The other is I wanted to check the exact values each function passes to bind function as in the description, which we can't know with the end-to-end tests.

I see now, thanks. The only thing which I am a bit concerned about is the complexity introduced by this change, in the sense that it is relying very much on the bind implementation of each expression. I think it would be great if we could generalize it.
Can we add a generic function similar to the createLambda in the analyzer for this?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2272/
Test PASSed.

Test build #94887 has started for PR 22131 at commit dec6978.

Test build #94888 has started for PR 22131 at commit 22c3f48.

sorry, I wasn't very specific in my previous comment. I meant adding a method like:
def validateBinding(e: HigherOrderFunction): Expression = {
    e.bind { case (f: LambdaFunction, dataTypes) =>
      f.arguments.zip(dataTypes).foreach {
        case (arg, (dt, n)) =>
          assert(arg.dataType == dt)
          assert(arg.nullable == n)
      }
      f
    }
  }

and then validate all the expressions, so use it like:
validateBinding(ArrayTransform(....))

What do you think about this?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2274/
Test PASSed.

Test build #94883 has finished for PR 22131 at commit 8154bf4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94883/
Test PASSed.

LGTM

LGTM too

Test build #94887 has finished for PR 22131 at commit dec6978.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #94888 has finished for PR 22131 at commit 22c3f48.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94888/
Test PASSed.

Test build #94891 has finished for PR 22131 at commit 8a844b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94891/
Test PASSed.

@mgaido91 @mn-mikke On second thought, how about this?
If you don't like it, I'll revert it soon.

I preferred the previous one (but yes, definitely good to add the check about the args size), but I am fine with this one too. Thanks @ueshin