SDK allows insertion of items with '/' in id, but does not allow delete

I am in the midst of a migration where 1 record out of 900 had a '/' in the identifier, it seems like the import process worked fine using the SDK, but then I can't actually look at the record via Cosmos Data Explorer or delete the item when I want to try to replace the item without the '/'.

Hi @onionhammer, this is currently an issue mostly on the fact that the service does not reject invalid characters. SDK does not apply any validation because it is not possible in all scenarios.
If you provide a Stream calling the Stream APIs for example, you are in control of that Stream content.
Any validation should be enforced on the service.

@ealsur So the service selectively enforces that restriction.. makes sense, is there a way to file a bug at the service?
Is there a way to delete an item with an invalid id without deleting the whole container?

This link has sample code for how to delete the document: https://docs.microsoft.com/en-us/azure/cosmos-db/troubleshoot-not-found

@ealsur So the service selectively enforces that restriction.. makes sense, is there a way to file a bug at the service?
Is there a way to delete an item with an invalid id without deleting the whole container?

The service is not applying any validation.
The problem is that your Read request fails because the URL the NodeJS SDK the Portal uses to fetch the item has the Id in it, and that is why it is broken. REST API specification of the service says you can read an item doing a GET to the /dbs/<yourdb>/colls/<yourcollection>/docs/<id>. And that breaks if the id contains any URL unsafe characters. So the request does not hit the backend with the value you intended, hence the 404.
Apparently, that is documented here: https://docs.microsoft.com/dotnet/api/microsoft.azure.documents.resource.id?view=azure-dotnet#remarks
I am only aware of UserVoice as the place to issue service requests: https://feedback.azure.com/forums/263030-azure-cosmos-db

Interesting.. couldnt the {id} component be url encoded by the SDK?

That would mean the REST contract would need to change and REST consumers would also need to escape the ids and the service cannot tell when it is encoded and when it is not. It's not a matter of the SDK doing it or not, it's a matter of, that should have been the design from the start, adding it now I don't believe it's possible.
In any case, it is something that should be driven by the service REST API.

Btw, this issue is a duplicate of https://github.com/Azure/azure-cosmos-dotnet-v3/issues/1908

I can't see how it wouldn't be possible to return a bad request if an invalid id is being inserted (by the service, not the sdk) but I'll just accept it

True, I don't say it's not possible. The service could add a validation and reject any document with an id value that has those characters. That would be the ideal solution to this problem because users can save documents through any means that call the REST API (official SDKs, a REST request, custom built SDKs by third-parties), so adding the check on the SDK is not practical and would not solve the problem because it does not cover all the ways documents can be inserted.

@ealsur @FabianMeiswinkel Does this sample not work if the partition key path is /id ?
https://docs.microsoft.com/en-us/azure/cosmos-db/troubleshoot-not-found

OK, here's a workaround for any future user who comes across this; deleting items with invalid IDs appears to be a major pain in the ass, the only way I was able to do it was by invoking the API using the http client:
string GenerateMasterKeyAuthorizationSignature(string verb, string resourceId, string resourceType, string pk, string key, string keyType, string tokenVersion)
{
    var hmacSha256 = new System.Security.Cryptography.HMACSHA256 { Key = Convert.FromBase64String(key) };

    string payLoad = string.Format(System.Globalization.CultureInfo.InvariantCulture, "{0}\n{1}\n{2}\n{3}\n{4}\n",
            verb.ToLowerInvariant(),
            resourceType.ToLowerInvariant(),
            resourceId,
            utc_date.ToLowerInvariant(),
            pk
    );

    byte[] hashPayLoad = hmacSha256.ComputeHash(System.Text.Encoding.UTF8.GetBytes(payLoad));
    string signature = Convert.ToBase64String(hashPayLoad);

    return System.Web.HttpUtility.UrlEncode(String.Format(System.Globalization.CultureInfo.InvariantCulture, "type={0}&ver={1}&sig={2}",
        keyType,
        tokenVersion,
        signature));
}

var client = new System.Net.Http.HttpClient();

utc_date = DateTime.UtcNow.ToString("r").ToLowerInvariant();

//EXECUTE delete
var docId = "VE8FAKg1aiLWAAAAAAAAAA==".ToLowerInvariant(); // _rid in the document you want to delete
var verb = "delete";
var resourceType = "docs";
var resourceLink = "dbs/VE8FAA==/colls/VE8FAKg1aiI=/docs/VE8FAKg1aiLWAAAAAAAAAA==/"; // _self in the document you want to delete
var resourceId =  docId;
var pk = "your-documents-pk";

var authHeader = GenerateMasterKeyAuthorizationSignature(
    verb, resourceId, resourceType, "", cosmosKey, "master", "1.0");

client.DefaultRequestHeaders.Add("x-ms-date", utc_date);
client.DefaultRequestHeaders.Add("x-ms-version", "2018-12-31");
client.DefaultRequestHeaders.Add("authorization", authHeader);
client.DefaultRequestHeaders.Add("x-ms-documentdb-partitionkey", JsonConvert.SerializeObject(new [] { pk }));

var path = new Uri(baseUri: new ("https://your-cosmos-account.documents.azure.com:443"), relativeUri: resourceLink);
var response = await client.DeleteAsync(path);
await response.Content.ReadAsStringAsync()


How to add more email addresses in the rule

I want to add more email addresses at the location marked in the picture, but I don't know how to do it

You have to duplicate the whole block of rule


Is this desktop Outlook only?

I can add the extension in this example and I see it "my add-ins", but it doesn't do anything. I'm using https://outlook.live.com/mail/inbox, and I don't see a Home tab or a Show Taskpane option. Am I supposed to use the Desktop Outlook version for this example?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 15be7148-d1e8-b58b-b026-9f072efb638a
Version Independent ID: 53a5d00c-d82b-6536-2e80-3e2a3452879a
Content: Build your first Outlook add-in - Outlook Developer
Content Source: docs/add-ins/quick-start.md
Product: outlook
Technology: add-ins
GitHub Login: @o365devx
Microsoft Alias: o365devx

@giannif Thanks for asking about this. The quick start should work on the various clients, including the browser. Can you take a look at the Outlook on the web screenshot in the tutorial to see if that shows how to access the add-in? If it does, I'll update the quick start with a similar screenshot.
Thanks.

@ElizabethSamuel-MSFT it does not show how to access the add in. The UI looks completely different than what I have on https://outlook.live.com/mail/inbox

So you do the quick start guide, and the add in is added successfully, but there are no visible UI elements in Outlook

@giannif Thanks for sharing a screenshot.
To access the add-ins, you need to click the "..." at the end of the "Send" button row. That is how you access additional/overflow commands.

@ElizabethSamuel-MSFT yes, I tried that right away. The add in does not appear:

That's why I thought it was maybe Outlook Desktop only, because in the tutorial, there is no "..." button to click, the add in appears in a different place all together:

@giannif I recommend you try the sideloading instructions for new/modern Outlook and let us know if that helps.

@ElizabethSamuel-MSFT I am able to add the add in, that is not the issue.

The add in just doesn't do anything and never appears in the user interface under any menu I can find. I've clicked every "..."

@giannif Can you select an existing message (versus composing a new message) and see if the add-in shows up there? I think the add-in is configured to work in an existing message.

No there is nothing there.

I'm no longer in need of help here exactly, since I have a compose extension working. Perhaps you can do the tutorial yourself and try to get it in https://outlook.live.com/mail instead of me taking screenshots. Should only take a few minutes to set up

@giannif I think that's the wrong menu. Do you have an overflow option on the message itself like in the following image?

I have this item on my backlog to update the various screenshots in the quick start and tutorial. I am glad that you got the compose extension point to work.

@ElizabethSamuel-MSFT wow, yep, there it is!! I never would have found that. New screenshots will help. I also think the add in itself can be improved. I still have no idea what the word Contoso even is or means. I'm not an Outlook user so I was extra lost


Feature/dataset landing page changes

What
https://trello.com/c/5EvP7odn/3089-amend-few-changes-on-the-landing-page
https://github.com/ONSdigital/dp-frontend-dataset-controller/pull/83 .  criteria 2
sidebar links size 14px
'Usage Information' to 'Usage information'
'Previous Versions'  to 'Previous version'
release text left aligned on phones
How to review
https://dev.cmd.onsdigital.co.uk/datasets/mid-year-pop-est/editions/time-series/versions/2
release text is (Release date:  Release frequency:  Next release:)
Who can review
crispin - jon

All looks good 👍


[WIP] Add inline transformations of several event types

See:https://github.com/JasperFx/marten/issues/404
I changed the api a bit, and you can also configure it from the initialize like this
 _.Events.Project()
    .Event<QuestStarted>((session, streamId, @event) => doSomethingWithTheEvent(@event))

Tests are running from IDE. I Couldn't run the global RAKE script due to dotnet versioncore issues on this dev machine. Will fix this soon, but let me know if you have problems running the tests on TC.

Updated the title just to indicate that it's not ready to pull yet.

@tim-cools I'm sorry, but could I talk you into rebasing this on master before I pull it in?

This looks awesome and is exactly what I need right now.
I do have a comment/question: should the Project() and InlineTransformation() methods be part of the ProjectionCollection instead of the EventGraph (which might change the method names slightly)? That would match the convention used by other inline projections like aggregates.
options.Events.InlineProjections.Project()
    .Event<QuestStarted>((session, streamId, evt) => events.Add(evt))
    .Event<MembersJoined>((session, streamId, evt) => events.Add(evt))
    .Event<MonsterSlayed>((session, streamId, evt) => events.Add(evt));
// renamed from InlineTransformation() to match AggregateStreamsWith<T>()
options.Events.InlineProjections.TransformStreamsWith(projection);

This actually bring up a broader point (which probably needs it's own issue or something) - as a new user I find the API around event projections to be a bit confusing. Especially with regards to the differences around aggregates, projections, and transformations. These terms seem to be used somewhat interchangeably, both in the core class/interface definitions and in the supporting registration methods. It's not clear from the outset why something might be an aggregate over a projection. Is one a superset of the other? As an example, I add an aggregate through a class called InlineProjections. These kinds of questions could probably be addressed with some kind of glossary in the help documents around projections or alternatively by taking a look at the API prior to 1.0 and refining some of the semantics.

Just realized that InlineTransformation() isn't actually part of this PR and was already there. The comment about moving it under the ProjectionCollection still stands but might be out of scope for this PR. I think that illustrates the second point about the API though. Why do I add an inline IProjection projection under the root EventGraph but then add an aggregate through a property called InlineProjections? The public event projection API just seems a little jumbled in it's current state.

@jeremydmiller @daveaglick Thanks for the feedback! I'm out of town but will look into this in the beginning of next week...

@tim-cools I looked closer at this one, and I'm honestly a little dubious. Having to specify a separate lambda for synchronous vs. asynchronous invocations is going to be a pretty serious drag.

@jeremydmiller I updated the PR.
I also felt that Having to specify a separate lambda for synchronous vs. asynchronous invocations is going to be a pretty serious drag.
So I updated the projection to be more intelligent. It now also fetches the projected view from the db. This is what I do in each projection anyway. Doing so the projection implementation only needs to update the projected view data. The (a)sync is only used to to load the projected view from db, so a single configuration can be used for both sync an async.
            StoreOptions(_ =>
            {
                _.AutoCreateSchemaObjects = AutoCreate.All;
                _.Events.ProjectView<PersistedView>()
                    .ProjectEvent<QuestStarted>((view, @event) => events.Add(@event))
                    .ProjectEvent<MembersJoined>(e => e.QuestId, (view, @event) => events.Add(@event))
                    .ProjectEvent<MonsterSlayed>(e => e.QuestId, (view, @event) => events.Add(@event));
            });

And like this if you want to use a class:

    public class PersistDocumentProjection : ViewProjection<PersistedView>
    {
        public PersistDocumentProjection()
        {
            ProjectEvent<QuestStarted>(Persist);
            ProjectEvent<MembersJoined>(e => e.QuestId, Persist);
            ProjectEvent<MonsterSlayed>(e => e.QuestId, Persist);
        }

        private void Persist<T>(PersistedView view, T @event)
        {
            view.Events.Add(@event);
        }
    }

Let me know if I broke some else as some not related tests are failing on my dev machine...
@daveaglick I noticed you had a discussion about the projection but had no time to get into the details. Let me know if I missed something important?

@tim-cools Thanks for the heads up! I had actually starting working on my own implementation based on this one as a starting point to play around with some edge cases I encountered while applying Marten to some LoB apps. Things like views based on events with multiple levels of indirection to the aggregate. I'm out of the office today, but I'll take a look tomorrow and see if the updates hit these use cases (or how I can help enhance the PR of not).

This is going to be a long one...sorry! I want to make sure I've properly documented the why and how of what I've been working on.
I've been playing with aggregate projection support for a few days, starting out with your PR, but expanding a bit to encompass the following requirements:

Projections should support both one-to-one and many-to-one event-to-view mappings (I didn't look at many-to-many since I was focused on aggregate projections).
A projection should be able to specify a view object based on each individual event.
Specifying the view object should be optional, and if one isn't specified a default behavior such as loading the view from the event stream ID should be used.

These are requirements that I actually have. Particularly the generation of view projections from multiple events that have several levels of indirection to the "main" aggregate root object. I ran into these types of scenarios pretty quickly when experimenting with porting a complex line of business application.
An example that might go with the whole "quest" domain is to consider that each Party has Person members, each Person can carry one or more named Weapon, and that there are also Tavern objects where a Party can dine. Now suppose you want to create a read-side view projection from various events listing the name of all of the weapons present at a given tavern. A Weapon belongs (references) a Person, which in turn belongs to a Party, which dines at the Tavern. If I process an event like WeaponRenamed, I'll need to follow the associations all the way up that chain to construct my view projection at event processing time.
@jeremydmiller suggested I take a look at using the existing support for this, possibly by switching out the AggregateFinder<T> implementation in a AggregationProjection<T>. The main challenge with this is that AggregateFinder<T> assumes the aggregate view is the same for all the events in a given stream (I.e., it takes the view ID from the stream ID).
Anyway, here's what I came up with. This is from LINQPad so it can be executed as a test. The actual Projection<TView> class in near the bottom.

IProjection implementation and example/test code
```
void Main()
{
	DocumentStore store = DocumentStore.For(x =>
	{
		x.Connection("host=localhost;database=marten_test;password=postgres;username=postgres");
		x.AutoCreateSchemaObjects = AutoCreate.All;
		x.Logger(new NulloMartenLogger());
	// Set up events
	x.Events.DatabaseSchemaName = "events";
	x.Events.AddEventType(typeof(AAdded));
	x.Events.AddEventType(typeof(BAdded));
	x.Events.AddEventType(typeof(CAdded));
	x.Events.AddEventType(typeof(CNameChanged));
	x.Events.InlineProjections.AggregateStreamsWith<A>();
	x.Events.InlineProjections.AggregateStreamsWith<B>();
	x.Events.InlineProjections.AggregateStreamsWith<C>();
	x.Events.InlineProjections.Add(new CNamesByAProjection());
});
store.Advanced.Clean.CompletelyRemoveAll();
store.Advanced.Clean.DeleteAllDocuments();

// Create some As
Guid a1Id = Guid.NewGuid();
$"a1: {a1Id}".Dump();
Guid a2Id = Guid.NewGuid();
$"a2: {a2Id}".Dump();
using (var session = store.OpenSession())
{
	session.Events.StartStream<A>(a1Id, new AAdded
	{
		Id = a1Id,
		Name = "A1"
	});
	session.Events.StartStream<A>(a2Id, new AAdded
	{
		Id = a2Id,
		Name = "A2"
	});
	session.SaveChanges();
}

// Create some Bs
Guid a1b1Id = Guid.NewGuid();
$"a1b1: {a1b1Id}".Dump();
Guid a1b2Id = Guid.NewGuid();
$"a1b2: {a1b2Id}".Dump();
Guid a2b1Id = Guid.NewGuid();
$"a2b1: {a2b1Id}".Dump();
using (var session = store.OpenSession())
{
	session.Events.StartStream<B>(a1b1Id, new BAdded
	{
		Id = a1b1Id,
		AId = a1Id
	});
	session.Events.StartStream<B>(a1b2Id, new BAdded
	{
		Id = a1b2Id,
		AId = a1Id
	});
	session.Events.StartStream<B>(a2b1Id, new BAdded
	{
		Id = a2b1Id,
		AId = a2Id
	});
	session.SaveChanges();
}

// Create some Cs
Guid a1b1c1Id = Guid.NewGuid(); // Blue
$"a1b1c1: {a1b1c1Id}".Dump();
Guid a1b1c2Id = Guid.NewGuid(); // Red
$"a1b1c2: {a1b1c2Id}".Dump();
Guid a1b2c1Id = Guid.NewGuid(); // Green
$"a1b2c1: {a1b2c1Id}".Dump();
Guid a2b1c1Id = Guid.NewGuid(); // Yellow
$"a2b1c1: {a2b1c1Id}".Dump();
Guid a2b1c2Id = Guid.NewGuid(); // Purple
$"a2b1c2: {a2b1c2Id}".Dump();
using (var session = store.OpenSession())
{
	session.Events.StartStream<C>(a1b1c1Id, new CAdded
	{
		Id = a1b1c1Id,
		BId = a1b1Id,
		Name = "Blue"
	});
	session.Events.StartStream<C>(a1b1c2Id, new CAdded
	{
		Id = a1b1c2Id,
		BId = a1b1Id,
		Name = "Red"
	});
	session.Events.StartStream<C>(a1b2c1Id, new CAdded
	{
		Id = a1b2c1Id,
		BId = a1b2Id,
		Name = "Green"
	});
	session.Events.StartStream<C>(a2b1c1Id, new CAdded
	{
		Id = a2b1c1Id,
		BId = a2b1Id,
		Name = "Yellow"
	});
	session.Events.StartStream<C>(a2b1c2Id, new CAdded
	{
		Id = a2b1c2Id,
		BId = a2b1Id,
		Name = "Purple"
	});
	session.SaveChanges();
}

// Get the current projection
using (var session = store.OpenSession())
{
	CNamesByA[] results = session.Query<CNamesByA>().ToArray();
	Debug.Assert(results.First(x => x.Id == a1Id).CNamesByC
		.Contains(new KeyValuePair<Guid, string>(a1b1c1Id, "Blue")));
	Debug.Assert(results.First(x => x.Id == a1Id).CNamesByC
		.Contains(new KeyValuePair<Guid, string>(a1b1c2Id, "Red")));
	Debug.Assert(results.First(x => x.Id == a1Id).CNamesByC
		.Contains(new KeyValuePair<Guid, string>(a1b2c1Id, "Green")));
	Debug.Assert(results.First(x => x.Id == a2Id).CNamesByC
		.Contains(new KeyValuePair<Guid, string>(a2b1c1Id, "Yellow")));
	Debug.Assert(results.First(x => x.Id == a2Id).CNamesByC
		.Contains(new KeyValuePair<Guid, string>(a2b1c2Id, "Purple")));
	results.Dump();
}

// Change a couple names
using (var session = store.OpenSession())
{
	session.Events.Append(a1b1c2Id, new CNameChanged
	{
		Name = "Red2"
	});
	session.Events.Append(a2b1c1Id, new CNameChanged
	{
		Name = "Yellow2"
	});
	session.SaveChanges();
}

// Get the current projection
using (var session = store.OpenSession())
{
	CNamesByA[] results = session.Query<CNamesByA>().ToArray();
	Debug.Assert(results.First(x => x.Id == a1Id).CNamesByC
		.Contains(new KeyValuePair<Guid, string>(a1b1c1Id, "Blue")));
	Debug.Assert(results.First(x => x.Id == a1Id).CNamesByC
		.Contains(new KeyValuePair<Guid, string>(a1b1c2Id, "Red2")));
	Debug.Assert(results.First(x => x.Id == a1Id).CNamesByC
		.Contains(new KeyValuePair<Guid, string>(a1b2c1Id, "Green")));
	Debug.Assert(results.First(x => x.Id == a2Id).CNamesByC
		.Contains(new KeyValuePair<Guid, string>(a2b1c1Id, "Yellow2")));
	Debug.Assert(results.First(x => x.Id == a2Id).CNamesByC
		.Contains(new KeyValuePair<Guid, string>(a2b1c2Id, "Purple")));
	results.Dump();
}

}
// Aggregates
public class A
{
public Guid Id { get; set; }
public string Name { get; set; }
public void Apply(AAdded added)
{
	Id = added.Id;
	Name = added.Name;
}

}
public class B
{
public Guid Id { get; set; }
public Guid AId { get; set; }
public void Apply(BAdded added)
{
	Id = added.Id;
	AId = added.AId;
}

}
public class C
{
public Guid Id { get; set; }
public Guid BId { get; set; }
public string Name { get; set; }
public void Apply(CAdded added)
{
	Id = added.Id;
	BId = added.BId;
	Name = added.Name;
}

public void Apply(CNameChanged nameChanged)
{
	Name = nameChanged.Name;
}

}
// Events
public class AAdded
{
public Guid Id { get; set; }
public string Name { get; set; }
}
public class BAdded
{
public Guid Id { get; set; }
public Guid AId { get; set; }
}
public class CAdded
{
public Guid Id { get; set; }
public Guid BId { get; set; }
public string Name { get; set; }
}
public class CNameChanged
{
public string Name { get; set; }
}
// Projections
public class CNamesByA
{
public Guid Id { get; set; }
public Dictionary<Guid, string> CNamesByC { get; set; } = new Dictionary<Guid, string>();
}
public class CNamesByAProjection : Projection
{
public CNamesByAProjection()
{
	Apply<CAdded>(
		(evt, view) => view.CNamesByC[evt.Data.Id] = evt.Data.Name,
		(session, stream, evt) => session.Load<B>(evt.Data.BId).AId);
	Apply<CNameChanged>(
		(evt, view) => view.CNamesByC[evt.StreamId] = evt.Data.Name,
		(session, stream, evt) => session.Load<B>(session.Load<C>(evt.StreamId).BId).AId);
}

}
// IProjection implementation
public class Projection : IProjection where TView : class, new()
{
private readonly IDictionary<Type, Transform> _transforms
= new ConcurrentDictionary<Type, Transform>();
public Type Produces => typeof(TView);

public Type[] Consumes => _transforms.Keys.ToArray();

public AsyncOptions AsyncOptions { get; } = new AsyncOptions();

public Projection<TView> Apply<TEvent>(Action<Event<TEvent>, TView> transformation, Func<IDocumentSession, EventStream, Event<TEvent>, Guid> getViewId)
{
	if (transformation == null)
	{
		throw new ArgumentException(nameof(transformation));
	}
	if (getViewId == null)
	{
		throw new ArgumentException(nameof(getViewId));
	}

	_transforms.Add(typeof(TEvent), new Transform
	{
		Transformation = (evt, view) => transformation((Event<TEvent>)evt, view),
		GetViewId = (session, stream, evt) => getViewId(session, stream, (Event<TEvent>)evt)
	});

	return this;
}

public Projection<TView> Apply<TEvent>(Action<Event<TEvent>, TView> transformation, Func<EventStream, Event<TEvent>, Guid> getViewId) =>
	Apply(transformation, (session, stream, evt) => getViewId(stream, evt));

public Projection<TView> Apply<TEvent>(Action<Event<TEvent>, TView> transformation) =>
	Apply(transformation, (session, stream, evt) => stream.Id);

void IProjection.Apply(IDocumentSession session, EventStream[] streams)
{
	// First get all the view IDs, caching the event ID to view ID map
	List<EventToTransform> eventsToTransform = GetEventsToTransform(session, streams);

	// Load all the views
	session.LoadMany<TView>(eventsToTransform.Select(x => x.ViewId).Distinct().ToArray());

	// Apply the events, creating new views when needed
	// Make sure to cache which views have already been added since we may operate on the same view more than once
	HashSet<Guid> viewIds = new HashSet<System.Guid>();
	foreach (EventToTransform eventToTransform in eventsToTransform)
	{
		TView view = session.Load<TView>(eventToTransform.ViewId)
			?? GetNewView(session, eventToTransform.ViewId);
		eventToTransform.Transform.Transformation(eventToTransform.Event, view);
		if (viewIds.Add(eventToTransform.ViewId))
		{
			session.Store(view);
		}
	}
}

async Task IProjection.ApplyAsync(IDocumentSession session, EventStream[] streams, CancellationToken token)
{
	// First get all the view IDs, caching the event ID to view ID map
	List<EventToTransform> eventsToTransform = GetEventsToTransform(session, streams);

	// Load all the views
	await session.LoadManyAsync<TView>(token, eventsToTransform.Select(x => x.ViewId).Distinct().ToArray()).ConfigureAwait(false);

	// Apply the events, creating new views when needed
	// Make sure to cache which views have already been added since we may operate on the same view more than once
	HashSet<Guid> viewIds = new HashSet<System.Guid>();
	foreach (EventToTransform eventToTransform in eventsToTransform)
	{
		TView view = await session.LoadAsync<TView>(eventToTransform.ViewId, token).ConfigureAwait(false)
			?? GetNewView(session, eventToTransform.ViewId);
		eventToTransform.Transform.Transformation(eventToTransform.Event, view);
		if (viewIds.Add(eventToTransform.ViewId))
		{
			session.Store(view);
		}
	}
}

private List<EventToTransform> GetEventsToTransform(IDocumentSession session, EventStream[] streams)
{
	List<EventToTransform> eventsToTransform = new List<EventToTransform>();
	foreach (EventStream stream in streams)
	{
		Transform transform = null;
		foreach (IEvent evt in stream.Events.Where(evt => _transforms.TryGetValue(evt.Data.GetType(), out transform)))
		{
			Guid viewId = transform.GetViewId(session, stream, evt);
			eventsToTransform.Add(new EventToTransform
			{
				Stream = stream,
				Event = evt,
				Transform = transform,
				ViewId = viewId
			});
		}
	}
	return eventsToTransform;
}

private TView GetNewView(IDocumentSession session, Guid id)
{
	TView view = new TView();
	session.DocumentStore.Schema.IdAssignmentFor<TView>().Assign(view, id);
	return view;
}

private class EventToTransform
{
	public EventStream Stream { get; set; }
	public IEvent Event { get; set; }
	public Transform Transform { get; set; }
	public Guid ViewId { get; set; }
}

private class Transform
{
	public Action<IEvent, TView> Transformation { get; set; }
	public Func<IDocumentSession, EventStream, IEvent, Guid> GetViewId { get; set; }
}

}
</details>

This is similar, though has also diverged a bit, from this PR. Some key differences:
* There is a single handler for each event instead of splitting into two
* This does two "passes", the first to get all the view IDs from the events and then to fetch all the views in bulk (which may or may not trigger loads from the database to get the correct view ID) and then to apply the events to the views

TL;DR: The main thing I'd request is that the view ID selector function in the PR be passed the `IDocumentSession` so that you can get a view that requires loading additional documents to figure out it's ID. The rest looks like it mostly coding style or implementation details.

I can also see where there's a potential to combine this code with the existing aggregate code. There's really no reason there should be multiple APIs for generating aggregates. One API that is general enough to support different cases (like using in-view `Apply()` methods vs. a fluent API) should cover it. Maybe I'll take a look at some consolidation strategies once this one is accepted.

@daveaglick sure, sound like a good use case.
I also like how you end up with only one handler collection, I try to incorporate that too.
I didn't use Apply as method name, because it doesn't apply the event, but it configures how the event will be projected. Hence the ProjectEvent name. But I'm open for discussion..
@jeremydmiller
One scenario that is not covered yet is the case where you have multiple view types per projection. This is a better alternative to using projections with dependencies to other dependencies.
Base Projection --> Dependent Projection
If you have projection (dependent) that depends on another projection (base) you have to replay both in case you change the base projection. This replay mechanism can become quit complicated if you have multiple dependencies in you views.
What you can do instead is let the projection store all necessary data in its own views. If you do so you only need to replay the single projection in case somethings changes. And you don't have the complicated dependencies. You do end up with more duplicated data in the db, but data storage is cheap anyway.

I updated the PR with the overload that accepts the DocumentSession for selecting the id as requested by @daveaglick . I also had to refactor the ViewProjection to support this. This is loosely based on daves work.

Looking good, thanks! I'll try swapping out mine for this in my own tests and see if I hit any snags. :thumbsup:

:sparkles: :shipit:


(Dev task) Create Basic Keeper Export/Import Genesis for Rewards module and corresponding keeper methods

Summary
Create Keeper for Rewards module genesis keeper.  Implement all required keeper functionality:

Create Key Types
Create Iterators
Create CRUD operations

Problem Definition

Proposal

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

this is done


WIP - DO NOT MERGE: Fixes #912: update WFS version

@Ladicek I've updated the WildFly Swarm version attribute to show the Productized version. WDYT?

This is wrong IMHO. One, it doesn't affect the WildFly Swarm guide, and two, it's used in URLs to Swarm community docs.

@Ladicek I believe the change does indeed affect the WildFly Swarm guide (Specifically the chapter that deals with using the BOM). The version displayed in the guide is sourced from the variable that has been changed.
I have completely missed the URLs, though, so thank you for pointing that out:) I'll fix those and update the PR.

I think this breaks the rules of how WildFly Swarm docs should be updated, but is OK for now. @tradej will need to fix this properly when he returns :-)

thanks @Ladicek.
merging

merged: 999fc44ec7a6e3dc2b63d8b84b284d19f9c7e94f

@lthon Thank you:) FTR, I'd personally be more comfortable letting Sheldon handle this, too... But as an emergency fix, I hope this'll do:)

Hi, just to confirm, I believe this fix is perfectly fine and can stay as a permanent one. We both want the product version (now stored in WildFlySwarmProductVersion) and the community version (now stored in WildFlySwarmVersion, but it can be renamed to, say, WildFlySwarmCommunityVersion).
As for the rules of updating, since no files in docs/topics/wildfly-swarm were changed, this PR is 100% by the book.
As for setting the WFS product version in document-attributes.adoc, I'm afraid we'll have to do it because we can not reproduce a WFS product build completely (setting the version happens there), so this is acceptable.

I knew there is a directory with wildfly-swarm in the name that must not be updated manually, and probably some brain hiccup caused me to believe that this PR did that. Happy to have been wrong :-)

I appreciate you remembered that; it's better than if it was forgotten. :)


Use pg_upgradecluster --method=upgrade --link

Testing Plan: Local upgrades from 10 -> 12

From https://github.com/zulip/zulip/pull/15721#issuecomment-656290049.

Merged as 5ff302541128fcfefe5ce43a84c55d3551f56837 with no changes, thanks @alexmv!


World bans

i want diffrent bans i diffrent worlds like i have a hardcore world and one survival world, but when they die in the hardcore world i only want them to be banned i that world not the survival world. any tips on how to do that?

Hi, I know this is 5 years late, but the best solution for this is likely

Manually denying their permission to that world when you want (for example, on death)
Searching for a plugin to customize what happens on ban events
Having another plugin manage the "ban" per world, I.E doing #1 automatically
Try the plugin or bungeecord option as suggested by ghost.


Phar throwing errors in PHP 8.0.

PHPMD version: 2.10.0
PHP Version: 8.0.3
Installation type: phar
Operating System / Distribution & Version: macOS

Current Behavior
I'm seeing multiple errors during a single lint execution.
./Bin/phpmd Test.php json cleancode,codesize,controversial,design,naming,unusedcode
PHP Deprecated:  Function libxml_disable_entity_loader() is deprecated in phar:///<xxxxx>/Bin/phpmd/vendor/symfony/config/Util/XmlUtils.php on line 54

Deprecated: Function libxml_disable_entity_loader() is deprecated in phar:///<xxxxx>/Bin/phpmd/vendor/symfony/config/Util/XmlUtils.php on line 54
PHP Deprecated:  Function libxml_disable_entity_loader() is deprecated in phar:///<xxxxx>/Bin/phpmd/vendor/symfony/config/Util/XmlUtils.php on line 68

Deprecated: Function libxml_disable_entity_loader() is deprecated in phar:///<xxxxx>/Bin/phpmd/vendor/symfony/config/Util/XmlUtils.php on line 68
PHP Deprecated:  Function libxml_disable_entity_loader() is deprecated in phar:///<xxxxx>/Bin/phpmd/vendor/symfony/dependency-injection/Loader/XmlFileLoader.php on line 517

Deprecated: Function libxml_disable_entity_loader() is deprecated in phar:///<xxxxx>/Bin/phpmd/vendor/symfony/dependency-injection/Loader/XmlFileLoader.php on line 517
PHP Deprecated:  Function libxml_disable_entity_loader() is deprecated in phar:///<xxxxx>/Bin/phpmd/vendor/symfony/dependency-injection/Loader/XmlFileLoader.php on line 519

Deprecated: Function libxml_disable_entity_loader() is deprecated in phar:///<xxxxx>/Bin/phpmd/vendor/symfony/dependency-injection/Loader/XmlFileLoader.php on line 519
{
    "version": "2.10.0snapshot202104261849",
    "package": "phpmd",
    "timestamp": "2021-04-27T02:27:17+00:00",
    "files": [
        {
            "file": "\/Users\/mike\/Developer\/phpmd.novaextension\/Test.php",
            "violations": [
                {
                    "beginLine": 3,
                    "endLine": 6,
                    "package": "+global",
                    "function": null,
                    "class": "Testasfsdfsdafsdfsadfsdfsdfsdafsadfsafsdfsdfdsfdsfsdaf",
                    "method": null,
                    "description": "Avoid excessively long class names like Testasfsdfsdafsdfsadfsdfsdfsdafsadfsafsdfsdfdsfdsfsdaf. Keep class name length under 40.",
                    "rule": "LongClassName",
                    "ruleSet": "Naming Rules",
                    "externalInfoUrl": "https:\/\/phpmd.org\/rules\/naming.html#longclassname",
                    "priority": 3
                }
            ]
        }
    ]
}

Expected Behavior

The errors make it difficult to parse the results programatically.
I would expect libxml_disable_entity_loader() not to be referenced when running in PHP 8.

Steps To Reproduce:
Run the above command on the following file:
<?php

class Testasfsdfsdafsdfsadfsdfsdfsdafsadfsafsdfsdfdsfdsfsdaf
{
    //
}

Checks before submitting

[X] Be sure that there isn't already an issue about this. See: Issues list
[X] Be sure that there isn't already a pull request about this. See: Pull requests
[X] I have added every step to reproduce the bug.
[X] If possible I added relevant code examples.
[X] This issue is about 1 bug and nothing more.
[X] The issue has a descriptive title. For example:  "JSON rendering failed on Windows for filenames with space".

Hello, the PHAR is a unique bundle, it's the exact same code for any PHP version, so we have to build using the lowest supported PHP version to ensure it works everywhere which prevents having latest Symfony packages in the PHAR version.
To get the latest dependencies, you need to use the composer installation. We'll get rid of old PHP version on the version 3.0. Optionally you can fork https://github.com/phpmd/phpmd/blob/master/.github/workflows/generate_phar.yml#L22 to build a PHAR for an upper version.
We could consider catching those deprecations but fixing them is not on our side.
Thanks.

Hello, the PHAR is a unique bundle, it's the exact same code for any PHP version, so we have to build using the lowest supported PHP version to ensure it works everywhere which prevents having latest Symfony packages in the PHAR version.

Well, you're contradicting yourself: the current PHAR indeed does not work everywhere (it's not working with PHP 8, as stated above).

Incorrect, libxml_disable_entity_loader() works in PHP 8, it's only deprecated.
And I'm answering the statement:

I would expect libxml_disable_entity_loader() not to be referenced when running in PHP 8.

It's only deprecated means that running phpmd will exit with a code different from 0. That's enough to say that it does not work.

Expecting a PHAR build to contain a different code in PHP 5, 7 or 8 is not possible, it involves to provide multiple PHAR and require user to switch from one to an other on each environment when the version may be different, we won't do that.

I guess that it's simple enough as adding a conditional based on PHP version in the code.

Are you suggesting we should already be compatible with PHP 12 too? 😆

I guess that it's simple enough as adding a conditional based on PHP version in the code.

That's exactly how we work around incompatibility across versions.

Also keep in mind that libxml_disable_entity_loader() is called from symfony/dependency-injection, it's not in our own library code.

Also keep in mind that libxml_disable_entity_loader() is called from symfony/dependency-injection, it's not in our own library code.

You're right about this.
Anyway, I don't get why you don't want to provide PHARs for different versions, almost every library does that.
Can't you provide a single PHAR (like now) and versioned PHARs (for such edge cases)?

running phpmd will exit with a code different from 0
No, it will exit with 0 (deprecation as you can see above are not fatal errors, process continues)

Also it's a user choice to report or not:
php -d error_reporting=24575 phpmd.phar folder json ...

almost every library does that.
What libraries are you referring to? I think you may be mixing providing different libraries versions having different PHP support ranges (we have the 3.x branch which will drop PHP 5) with 1 version of a library that split into multiple PHARs: and for the last, I it's certainly not standard.

Users can now work around this issue easily:

with php -d error_reporting=24575
using composer instead of the phar
build a phar from source: see our workflow: https://github.com/phpmd/phpmd/blob/master/.github/workflows/generate_phar.yml how we make the phar is also part of the open-source code

So my position (and I don't talk for the whole team) is that it's fine to fix this in 3.0.0 when we'll drop old PHP versions and so will upgrade symfony/dependency-injection

This is from CircleCI:

Is logs/pmd.xml properly generated?

Is logs/pmd.xml properly generated?

XML it's generated in a previous step, that's returning exit code 0.
By the way, using php -d error_reporting=24575 does not provide any improvement: phpmd is exiting with code 3 nonetheless

XML it's generated in a previous step

The report file should be written down by phpmd with the summary of the violation, I don't get how it can be done in a previous step.

using php -d error_reporting=24575 does not provide any improvement: phpmd is exiting with code 3 nonetheless

That's actually mean you're facing a different issue. I suspect the 3 exit code is not related to this current issue regarding libxml_disable_entity_loader deprecation.

Ah, sorry I misinterpreted your previous question (and replied about configuration XML, not log one).
So, I checked the log and I'm getting an error about "Unexpected token: private" that is related to a promoted property.
So, it looks like phpmd is not supporting PHP 8 syntax 😕

That's what I though, I checked and the issue here returns 0 code when checking with echo $?
You're likely have one of the unsupported syntax listed above:
https://github.com/pdepend/pdepend/issues?q=is%3Aissue+is%3Aopen+label%3Aphp8.0
There remain very few of them regarding the whole list of new syntax support we already implemented:
https://github.com/pdepend/pdepend/issues?q=is%3Aissue+label%3Aphp8.0+is%3Aclosed
We'll gladly merge your pull-request if you're willing to fix it.
If it's not one of the known issues, please create a new one.
After that I will cleanup this issue to remain it focused on the deprecation notice issue. Fatal errors should not be handled there else we'll never keep the track of the proper priority of it.

Thank you @kylekatarnls
I commented on related issue


Calc2

``from sympy import *
import time
import os
x,y,z=symbols('x y z')
'''kindly help me improve this program anyway you can contact my whatsapp<PHONE_NUMBER>2, bugs abound in it'''
print('this program works best in a ipython notebook, your terminal might lack the means of displaying the text format its not a code bug!\n')
#time.sleep(5)
#os.system('clear')
print("ensure you have the following modules installed sympy,time,os")
#time.sleep(3)
#os.system('clear')
print('Please let all your variables be in "x" or "y" or "z" in case of multiple integrals ')
#time.sleep(5)
os.system('clear')
sym='use exp(x) for e^x , asin() for inverse sin and so on, use lowercase oo for infinity'
print(sym)
#time.sleep(6)
os.system('clear')
que = input('type your question--- ensure your question is in a format of f(x)=x2 + xsin(x), furthermore your question should contain any of the following markers just one of them not two! ::area ::definite_integral::  integral ::double_integral::definite_double_integral ::differentiate.  e.g find the area of the ....under f(x) = x**2+5 between [0,5] you can replace the "between keyword but just one word" find the integral of bla bla bla, find the definite_integral of bla bla between (2,6) or between (0,Pi/2), differentiate or gradient should be used for differentials albeit you would  find the numeric value yourself! hint!=> dont put space inbetween the function ::\n')
ev_que=que.lower().split()
def def_int_grt(function,lim1,lim2):
sol=Integral(function,(x,lim1,lim2))
sol_d=sol.doit()
print(f"integral=f{sol},  solution={sol_d}")
def int_grl(funtion):
sol=Integral(function,x)
sol_d=sol.doit()
print(f"integral={sol},  solution={sol_d}")
def def_mult_integral(function,lim1,lim2):
sol=Integral(function,(x,lim1,lim2),(y,lim1,lim2))
sol_d=sol.doit()
print(f"integral={sol},  solution={sol_d}")
def mult_integral(function):
sol=Integral(function,x,y)
sol_d=sol.doit()
print(f"integral=f{sol},  solution={sol_d}")
def diff(function):
sol=Derivative(function,x)
sol_d=sol.doit()
print(f"derivative=f{sol},  solution={sol_d}")
function=ev_que[(ev_que.index('f(x)')+2)]
def solve(ev_que):
if  'integral' in ev_que:
int_grl(function)
elif 'double_integral' in ev_que:
mult_integral(function)
elif 'differentiate' in ev_que:
    diff(function)
else:
    print('incorrect format')

def def_solve(ev_que):
if 'definite_integral' or 'area' in ev_que:
b=ev_que[(ev_que.index('f(x)')+4)]
lim1=int(b[1])
lim2=int(b[3])
def_int_grt(function,lim1,lim2)
elif 'definite_double_integral' in ev_que:
b=ev_que[(ev_que.index('f(x)')+4)]
lim1=int(b[1])
lim2=int(b[3])
def_mult_integral(function,lim1,lim2)
else:print('incorrect format')
base=['area','definite_integral','definite_double_integral']
cnt=2
for line in base:
if line in ev_que:
def_solve(ev_que)
elif line not in ev_que:
solve(ev_que)


collectInputs return type differs on iOS from those documented in the API docs

Describe the bug
When trying the workaround for iOS collectInputs bug to pass a dummy collectInputs parameter we had other issues, it look like SDK doesn't return the documented object types in the result. It returns an object that is as follows
When FormType.SELECTION used
{
  "selectionResult": {
    "skipped": false,
    "selection": "Custom Amount",
    "toggles": []
  }
}

When FormType.NUMERIC used
{
  "numericResult": {
    "skipped": false,
    "toggles": [],
    "numericString": "20"
  }
}

which is different from what the SDK types defined are i.e.
export interface ICollectInputsResults {
    collectInputResults?: ICollectInputsResult[];
    error: StripeError;
}
export interface ICollectInputsResult {
    skipped: boolean;
    email?: string;
    numericString?: string;
    phone?: string;
    selection?: string;
    signatureSvg?: string;
    text?: string;
    toggles: ToggleResult[];
}

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Stripe Terminal React Native SDK version

(e.g., ^0.0.1-beta.19)

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

hey @asher5 thanks a lot for flagging this issue. We are working on fixing this in the new release to align iOS and Android types, and it is scheduled for end of July.

We unified the types in the beta.23 release


Proof-of-concept prefetcher optimization for some non-sequential read patterns

Description of change
Mountpoint prefetcher is built to maximize throughput on sequential reads. It works by downloading increasingly larger blocks of content in advance while Mountpoint receives sequential read requests. On out-of-order requests (outside a small tolerance window), a new download is started and the pre-fetched content is dropped.
One example of an access pattern where the current logic results in sub-optimal performance is a quasi-sequential pattern, where read requests are sequential for the most part, but can occasionally seek forward (i.e. skip some content) further than the current 16MiB tolerance window.
A small test using Mountpoint 1.4.1 on a c5.4xlarge instance in eu-west-1 records these timings:
Sequential reads (total<PHONE_NUMBER>2 bytes): 9.7s
Skip pattern, read 399MB, skip 99MB (total<PHONE_NUMBER> bytes): 44.4s

This PR demonstrates a change in the prefetcher which can significantly improve throughput for similar access patterns by not dropping in-flight downloads when they "cover" the forward seek. Once applied, the same test gives:
Skip pattern, read 399MB, skip 99MB (total<PHONE_NUMBER> bytes): 10.0s

Note that this is only a proof of concept to demonstrate the potential improvement. In order to adopt it in Mountpoint, it will require at least a few modifications, e.g.:

When examining in-flight GetObject requests, we should first look at the range already downloaded, rather than the whole requested range. One way to do it could be to have a "non-blocking" path to read/consume data only if already available.

config.max_forward_seek_distance could then be used in the heuristic to determine whether to wait for more data to be downloaded or cancel the request.

[!WARNING]
Not to be merged in its current form.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

Can you share the test code as well? Meaning the application that is reading the data through MP.

Superseded by #797


Self-hosted Linux CI doesn't find the Rust cache

Example workflow: https://github.com/lurk-lab/lurk-rs/actions/runs/6015221937/job/16316659175
The linux job successfully writes the Rust cache to the GitHub cloud, but doesn't find it on any subsequent run. It appears the cache key of v0-rust-linux-71783194-f226bfd3 is constant between workflows, but the action doesn't recognize the path due to the runner's unique workdir implemented in https://github.com/lurk-lab/gh-actions-runner/pull/11.
According to the GitHub Actions caching docs we could specify the path as an argument, but https://github.com/Swatinem/rust-cache doesn't seem to have this option.

@samuelburnham is this obviated by the switch to buildjet?

Yes, with a recent example at https://github.com/lurk-lab/lurk-rs/actions/runs/6496086292/job/17642298477.
The remaining self-hosted GPU actions wouldn't be able to use the cache anyway as they are triggered independently. Closing, thanks for the ping!


feat: fixes MGDCTRS-1511, update config add admin API authz using auth.redhat.com

This PR is waiting for the relevant code changes to be merged in k-f-m PR https://github.com/bf2fc6cc711aee1a0c2a/kas-fleet-manager/pull/1293


Drills and Connectors cargo

Hello. This calc is awesome.
To make it more accurate, it needs to consider drills and connectors, cuz they can carry a lot of cargo.
For example - that calc shows 92.5 tons of mass (1 large cargo + extra), but, in fact, fully loaded ship's mass is ~110tons (4 drills, 1 connector, 1 large cargo + same extra)

Thanks! Indeed, that would be nice. Unfortunately, I don't have a lot of time to work on the calculator as of late, so I'm not sure if I can add it any time soon. For now, you could approximate it by adding an extra medium cargo container on a small grid, or an extra small cargo container on a large grid, with "Any-fill with Ore" set to 100.

In a new experimental version of the calculator, drills and connectors are now supported: https://secalc2.gohla.nl/

Now not experimental any more and moved to https://secalc.gohla.nl/.


Build fails with cabana-0.3.0

    106    cd /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-build/examples && /home/junghans/spack.git/lib/spack/env/gcc/g++   -I/tmp/junghans/spack-stage/sp
            ack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-build/examples -I/tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/examples -I/tmp/
            junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src -I/tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-buil
            d/src -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/kokkos-3.1.01-ewfgo3wepuf7d37jgtinll7qk6zigof7/include -isystem /home/junghans/spack.git/opt/spack/linux-cent
            os7-haswell/gcc-8.2.0/openmpi-3.1.6-jnys4eardy2dlj25lklmxcxuarflps3i/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/cabana-0.3.0-qzqsu3nor3rltt3ecq63m6pgx
            xn7u7eb/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/cajita-0.1-lsqwyf355hijcigqrs6xwn24aarxxmd6/include -isystem /home/junghans/spack.git/opt/spack/lin
            ux-centos7-haswell/gcc-8.2.0/silo-4.10.2-ga3tsj5twrcwtk5zgeq6xapshedtoghj/include  -O2 -g -DNDEBUG   -march=core-avx2 -mtune=core-avx2 -pthread -o CMakeFiles/DamBreak.dir/dam_break.cpp.o -c /
            tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/examples/dam_break.cpp
     107    In file included from /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_ProblemManager.hpp:16,
     108                     from /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_Solver.hpp:15,
     109                     from /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/examples/free_fall.cpp:1:
     110    /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_ParticleCommunication.hpp: In function 'void ExaMPM::ParticleCommunication::redistrib
            ute(const LocalGridType&, int, ParticleContainer&, std::integral_constant<long unsigned int, CoordIndex>, bool)':
  >> 111    /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_ParticleCommunication.hpp:224:13: error: 'Distributor' is not a member of 'Cabana'
     112         Cabana::Distributor<device_type> distributor( local_grid.globalGrid().comm(),
     113                 ^~~~~~~~~~~

No idea, but what version of cabaa is needed to build this?

Distributor has been around for a while, it might just be missing an include to core/src/Cabana_Distributor.hpp?
I only see Cabana_Core.hpp here https://github.com/ECP-copa/ExaMPM/blob/master/src/ExaMPM_ParticleCommunication.hpp


[bug] cannot instantiate two connectivity-browser at the same time

as of d274c93ce38f3f1e1929da300235d1f8b62d3dcf or 055118579a26ad9e7454105f0d934d681f8cad00
to reproduce:
1/ insert the following code in any template:
    <connectivity-browser>
    </connectivity-browser>
    <connectivity-browser>
    </connectivity-browser>

expected behaviour: two connectivity browser renders
actual behaviour: only the first connectivity browser render
attached: DOM tree

Fixed


fix warnings reported by jshint

jshint reports minor warnings on generated parser code. still it makes troubles becasue I use jshint in git hook before commit.
thank you for this great tool

Sorry, but I won’t accept this change:

In case of continuation lines I  prefer operators to be at the beginning, not at the end.
The parser code as generated by PEG.buildParser is an expression, not a statement. Appending a ; at the end is wrong.

More generally, generated parser code can’t conform to any random JSHint settings its users use. The only guarantee is that it conforms to settings expressed in .jshintrc. If that doesn’t suite you, I recommend excluding the parser source from your JSHint checks.


WIP [RHOAIENG-7481] Add metrics columns to pipeline run table and modal param selector

Closes: RHOAIENG-7481
Description
Added action & modal to allow for metric related columns to be added to the pipeline runs table. The metric columns are added to local storage and unique for each experiment. The columns are re-orderable within the new modal.
NOTE: Some omissions for now due to lack of clarity:

Filtering by name within the modal - Still outstanding questions within Figma related to empty state
"Unavailable" icons shown instead of dashes for empty metric cells
It was suggested in Figma comments that we should auto-select 2 metrics if they exist as columns on behalf of the user. I have posed questions there expressing how this might be strange when page 1 has no metrics and following pages do have metrics. I think this is something that we can visit as a future enhancement.

When on a page that has no runs with metrics belonging to previously selected custom columns:

When max amount of columns are selected:

(cc @yannnz)
How Has This Been Tested?
Unit tests
Request review criteria:
Self checklist (all need to be checked):

[x] The developer has manually tested the changes and verified that the changes work
[x] Commits have been squashed into descriptive, self-contained units of work (e.g. 'WIP' and 'Implements feedback' style messages have been removed)
[ ] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[ ] The developer has added tests or explained why testing cannot be added (unit or cypress tests for related changes)

If you have UI changes:

[x] Included any necessary screenshots or gifs if it was a UI change.
[x] Included tags to the UX team if it was a UI/UX change (find relevant UX in the SMEs section).

After the PR is posted & before it merges:

[ ] The developer has tested their solution on a cluster by using the image produced by the PR to main

I wonder if we want a max height on this modal. it maxes out pretty quickly

@Gkrumbach07 I agree that is probably a bit too tall. I added a set content height of 500px for the body of the Modal. Let me know if you'd like it to be something different.


[Offline areas] Add bottom sheet to area selector with used/free/avail/required estimates

Potential layout:

@shobhitagarwal1612 FYI.

Consolidated into #481.


Making object storage optional

What this PR does / why we need it: This makes object storage optional. It mostly adds some config options to disable all backups.
Which issue this PR fixes (use the format fixes #<issue number>(, fixes #<issue_number>, ...) to automatically close the issue when PR gets merged): fixes #
Public facing documentation PR (if applicable)

Special notes for reviewer:
I have not fixed the tests yet. They should of course not give any error for a disabled feature.
Checklist:

[ ] Added relevant notes to WIP-CHANGELOG.md
[ ] Proper commit message prefix on all commits
[ ] Updated the public facing documentation
Is this changeset backwards compatible for existing clusters? Applying:

[ ] is completely transparent, will not impact the workload in any way.
[ ] requires running a migration script.
[ ] will create noticeable cluster degradation.
E.g. logs or metrics are not being collected or Kubernetes API server
will not be responding while upgrading.
[ ] requires draining and/or replacing nodes.
[ ] will change any APIs.
E.g. removes or changes any CK8S config options or Kubernetes APIs.
[ ] will break the cluster.
I.e. full cluster migration is required.

Really nice! Have you tested what happens if you have backups enabled with object storage set to none? Are you able to deploy any of the components that have backups?

If you have object storage as none but still have backups enabled, then helmfile will complain that some variables are not set (e.g. s3 endpoint) and throw an error.

Really nice! Have you tested what happens if you have backups enabled with object storage set to none? Are you able to deploy any of the components that have backups?

If you have object storage as none but still have backups enabled, then helmfile will complain that some variables are not set (e.g. s3 endpoint) and throw an error.

Will it do that for all components individually? Even for our harbor-backup chart?

Really nice! Have you tested what happens if you have backups enabled with object storage set to none? Are you able to deploy any of the components that have backups?

If you have object storage as none but still have backups enabled, then helmfile will complain that some variables are not set (e.g. s3 endpoint) and throw an error.

Will it do that for all components individually? Even for our harbor-backup chart?

I think so. I might have missed to test some of them.

Really nice! Have you tested what happens if you have backups enabled with object storage set to none? Are you able to deploy any of the components that have backups?

If you have object storage as none but still have backups enabled, then helmfile will complain that some variables are not set (e.g. s3 endpoint) and throw an error.

Will it do that for all components individually? Even for our harbor-backup chart?

I think so. I might have missed to test some of them.
edit: I might be wrong now that I think about it. Will do some testing

Did some new tests and no, this does not work as I thought it did.
So now I'm thinking that we might want to add things like this to our helmfile values:
#Example for harbor-backup.yaml.gotmpl
{{ if not (or (eq .Values.objectStorage.type "s3") (eq .Values.objectStorage.type "gcs") ) }}
{{ fail "Harbor backup requires s3 or gcs object storage, see Values.objectStorage.type" }}
{{ end }}

Though fail does not seem to work (or I'm using the wrong syntax. But do you think that this kind of validation is a good solution?

Did some new tests and no, this does not work as I thought it did.
So now I'm thinking that we might want to add things like this to our helmfile values:
#Example for harbor-backup.yaml.gotmpl
{{ if not (or (eq .Values.objectStorage.type "s3") (eq .Values.objectStorage.type "gcs") ) }}
{{ fail "Harbor backup requires s3 or gcs object storage, see Values.objectStorage.type" }}
{{ end }}

Though fail does not seem to work (or I'm using the wrong syntax. But do you think that this kind of validation is a good solution?

Yes, something like that would be nice


RFC: Rework the API [WIP]

Close #141
This PR modifies the API of cpal in order to not use futures anymore.
Using futures has several drawbacks. It makes the API more complex, forces the usage of Arcs/Mutexes in the backends, and could introduce semver issues in the future.
Additionally, if we wanted to make cpal compatible with tokio (so that they use the same architecture), we could only do that on Linux. The Win32 and MacOS backends use an API that is totally different from the sockets API of their platform, and therefore need their own events loop anyway.
The changes also remove the Voice struct. A voice is now simply a concept, and is referred by a VoiceId that can be cloned, compared and hashed. In other words, it's now more clear that the events loop owns the voices.

Link to the new API

I updated the MacOS backend, but I don't know if it's working.

I can confirm the CoreAudio backend works fine. Both examples run fine.

So for now the API only proposes a run() method that blocks the current thread entirely, and calls a callback whenever some data is needed.
This can be a bit annoying because it forces you to spawn a background thread, and makes the library annoying to use on emscripten who doesn't allow threads (who wants to block their whole program just for sound?).
I think in the future we should add some sort of detached_run function that, depending on the backend, either spawns a thread (Win32/Linux), does nothing (MacOS), or sets up a callback system (Emscripten). I wanted to give it a shot, but I realized that the details look more complicated than anticipated. I think that this should be postponed.

Probably worth mentioning that SDL_Audio's API doesn't expose a loop at all, it creates a thread or just delegates to AudioUnit executing a provided callback. detached_run makes sense to me, almost so far as only offering that.
Additionally, the surface area for the cpal abstraction layer could probably both allow for separate, delegate backends and a core-only mode now. It'd be cool to see cpal be usable on embedded systems that have DSPs but no libstd support.


master docs: fix Previous Releases empty href

@mdo where did you mean this link to point to?
https://github.com/twbs/bootstrap/blob/6335fbe224f28d2ce65c628756a84113ea65643f/site/layouts/partials/home/masthead.html#L24

Not @mdo, but I assume it's supposed to link to something like this?

Not @mdo, but I assume it'd point to a list of previous docs like this page, or an archive of previous releases?

Yeah, probably, I just want to double check.


App List returned on the portal config endpoint

Expected outcome

The get portal config by Id endpoint should return a list of  all applications related to a portal if the portal has no context types added. if the portal is configured with "contextTypes" the endpoint should return all global applications and context independent applications.
/api/portals/{portalId}

Business Value / Developer Experience
If context types are added to a portal the portal is considered  context driven and the frontend will the require a context selected before showing a  full list of applications.

User Acceptance Tests (UAT)

Test Case 1: Retrieve Portal Config Without Context Types
Given: A portal is configured without any context types.
When: A request is made to the endpoint /api/portals/{portalId} to retrieve the portal configuration by ID.
Then: The endpoint should return a list of all applications related to the portal.
And: The list should include both global applications and context-independent applications activated for the portal.
And: The response should not require any context selection.
Expected Outcome: The returned data should include a comprehensive list of applications that are associated with the portal, without any restrictions or filters based on context.
Test Case 2: Retrieve Portal Config With Context Types
Given: A portal is configured with one or more context types.
When: A request is made to the endpoint /api/portals/{portalId} to retrieve the portal configuration by ID.
Then: The endpoint should return a list of global applications and context-independent applications.
And: The list should exclude context-dependent applications, as these will only be shown when a context is selected.
And: The response should indicate that the portal is context-driven.
Expected Outcome: The returned data should include only global applications and context-independent applications. Context-dependent applications should not be present in the list.
Environments
All testing of issues are done in project-portal-test

project-portal-prod
project-portal-test
project-portal-feature

For Backend use these

project-portal-api-prod
project-portal-api-test
project-portal-api-feature

This may no longe be needed


⚠ ClamAV has degraded performance

In 524252b, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 75 ms

Resolved: ClamAV performance has improved in abad1be.


codegen generating incorrectly on v0.19.16

Trying to run codegen and it's generating code that references rest.HTTPClientFor which does not seem to exist in the sdk for this version. Is this a bug?
pkg/client/clientset/versioned/typed/scaling/v1alpha1/scaling_client.go:54:26: undefined: rest.HTTPClientFor
pkg/client/clientset/versioned/typed/scaling/v1alpha1/scaling_client.go:68:22: undefined: rest.RESTClientForConfigAndClient

go mod is asking for these:
	k8s.io/client-go v0.19.16
	k8s.io/code-generator v0.19.16

Full file being generated:
/*
Copyright West Coast DevOps

Author: Eric Uldall
Repo:   https://github.com/West-Coast-Devops/scheduled-scaler

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

// Code generated by client-gen. DO NOT EDIT.

package v1alpha1

import (
	"net/http"

	rest "k8s.io/client-go/rest"
	v1alpha1 "k8s.restdev.com/operators/pkg/apis/scaling/v1alpha1"
	"k8s.restdev.com/operators/pkg/client/clientset/versioned/scheme"
)

type ScalingV1alpha1Interface interface {
	RESTClient() rest.Interface
	ScheduledScalersGetter
}

// ScalingV1alpha1Client is used to interact with features provided by the scaling.k8s.restdev.com group.
type ScalingV1alpha1Client struct {
	restClient rest.Interface
}

func (c *ScalingV1alpha1Client) ScheduledScalers(namespace string) ScheduledScalerInterface {
	return newScheduledScalers(c, namespace)
}

// NewForConfig creates a new ScalingV1alpha1Client for the given config.
// NewForConfig is equivalent to NewForConfigAndClient(c, httpClient),
// where httpClient was generated with rest.HTTPClientFor(c).
func NewForConfig(c *rest.Config) (*ScalingV1alpha1Client, error) {
	config := *c
	if err := setConfigDefaults(&config); err != nil {
		return nil, err
	}
	httpClient, err := rest.HTTPClientFor(&config)
	if err != nil {
		return nil, err
	}
	return NewForConfigAndClient(&config, httpClient)
}

// NewForConfigAndClient creates a new ScalingV1alpha1Client for the given config and http client.
// Note the http client provided takes precedence over the configured transport values.
func NewForConfigAndClient(c *rest.Config, h *http.Client) (*ScalingV1alpha1Client, error) {
	config := *c
	if err := setConfigDefaults(&config); err != nil {
		return nil, err
	}
	client, err := rest.RESTClientForConfigAndClient(&config, h)
	if err != nil {
		return nil, err
	}
	return &ScalingV1alpha1Client{client}, nil
}

// NewForConfigOrDie creates a new ScalingV1alpha1Client for the given config and
// panics if there is an error in the config.
func NewForConfigOrDie(c *rest.Config) *ScalingV1alpha1Client {
	client, err := NewForConfig(c)
	if err != nil {
		panic(err)
	}
	return client
}

// New creates a new ScalingV1alpha1Client for the given RESTClient.
func New(c rest.Interface) *ScalingV1alpha1Client {
	return &ScalingV1alpha1Client{c}
}

func setConfigDefaults(config *rest.Config) error {
	gv := v1alpha1.SchemeGroupVersion
	config.GroupVersion = &gv
	config.APIPath = "/apis"
	config.NegotiatedSerializer = scheme.Codecs.WithoutConversion()

	if config.UserAgent == "" {
		config.UserAgent = rest.DefaultKubernetesUserAgent()
	}

	return nil
}

// RESTClient returns a RESTClient that is used to communicate
// with API server by this client implementation.
func (c *ScalingV1alpha1Client) RESTClient() rest.Interface {
	if c == nil {
		return nil
	}
	return c.restClient
}

Is there something we're missing here?

Testable fork/branch here: https://github.com/kirk-patton/scheduled-scaler/tree/v0.19.16

ah nvm, this was due to a path issue in the go folder so the new code was not being placed where we thought

I use<EMAIL_ADDRESS>   Also can't  find this  rest.HTTPClientFor.


removed non used parameter MPC_VELD_LP

This parameter added 5 years ago Here but it seems unused since.
Can someone confirm that? That I don't miss anything
By the way, the LP stand for Local position and not  Low Pass. as in its description

It's a bit confusing, but it is used. The name comes from the parent (MPC) + BlockDerivative (VELD) + BlockLowPass (LP).
https://github.com/PX4/Firmware/blob/25eca31e3a2c93f3b9d9b79121313d3c75280020/src/modules/mc_pos_control/mc_pos_control_main.cpp#L277
https://github.com/PX4/Firmware/blob/25eca31e3a2c93f3b9d9b79121313d3c75280020/src/modules/mc_pos_control/mc_pos_control_main.cpp#L281-L283
https://github.com/PX4/Firmware/blob/25eca31e3a2c93f3b9d9b79121313d3c75280020/src/lib/controllib/BlockDerivative.hpp#L69
https://github.com/PX4/Firmware/blob/25eca31e3a2c93f3b9d9b79121313d3c75280020/src/lib/controllib/BlockLowPass.hpp#L68
At this point I'd prefer to replace it with something a little more obvious in place. Other than LPE this is the only controllib usage in the code base.

@dagar, Thanks, I thought I missed something there...

Let me see if I can remove controllib entirely. https://github.com/PX4/Firmware/pull/15863


[POA-752] Mark old commands as deprecated

Marked old commands as deprecated, they will not show in help text now
might be removed for ec2 and kube
will be removed for daemon and nginx

Do we need to make these commands, unusable? Like error out like, apispec
https://github.com/akitasoftware/akita-cli/blob/ee18ae86fbe2d48b4483575eda7080205e5ef731/cmd/internal/apispec/cmd.go#L17

This can go as-is.
But thinking on this some more, how about if we removed the "will be removed" commands altogether, but leave their supporting code? Maybe also give the same treatment to commands that are already errors. I'm hoping this would be an easy blast-away-code operation. If you need to do surgery/refactoring of any kind to achieve this, then let's just go with what is here.

Removed old deprecated commands from root Cobra Command, but have kept their related code.


WIP: DataTables.jl Backport

The nl/datatables2 branch rebased with master.
TODO:

[X] Collect all the DataTable commits that should be backported (completed by @nalimilan).
[X] Rebase with master branch of DataFrames.jl
[X] Add fixes for issues identified during rebasing (e.g., reverted features, Compat wasn't removed in DataTables, still supports 0.5)
[ ] Remove DataFrames usage from DataStreams.jl?

NOTES:

This backport changes the default storage from DataArray to NullableArray.
We'll be porting another PR shortly after this which changes the default column storage to Vector{?T} now that Nulls.jl is registered.

Looks like the CI will still fail because of all the 0.7 deprecations on the current release of NullableArrays.jl.

Sorry I'm offline right now but I'll try to have a look next week.

Issues that (I think) can be closed with the merge of this PR
related to ModelMatrix:
https://github.com/JuliaStats/DataFrames.jl/issues/19
https://github.com/JuliaStats/DataFrames.jl/issues/319
https://github.com/JuliaStats/DataFrames.jl/issues/318
https://github.com/JuliaStats/DataFrames.jl/issues/246
https://github.com/JuliaStats/DataFrames.jl/issues/163
https://github.com/JuliaStats/DataFrames.jl/issues/60
https://github.com/JuliaStats/DataFrames.jl/issues/867
https://github.com/JuliaStats/DataFrames.jl/issues/823
https://github.com/JuliaStats/DataFrames.jl/issues/818
https://github.com/JuliaStats/DataFrames.jl/issues/788
https://github.com/JuliaStats/DataFrames.jl/issues/1018
related to (pooled)DataVectors
https://github.com/JuliaStats/DataFrames.jl/issues/31
https://github.com/JuliaStats/DataFrames.jl/issues/6
https://github.com/JuliaStats/DataFrames.jl/issues/329
https://github.com/JuliaStats/DataFrames.jl/issues/283
https://github.com/JuliaStats/DataFrames.jl/issues/448
https://github.com/JuliaStats/DataFrames.jl/issues/368
https://github.com/JuliaStats/DataFrames.jl/issues/895
https://github.com/JuliaStats/DataFrames.jl/issues/1114
https://github.com/JuliaStats/DataFrames.jl/issues/1002
read/writetable (can be moved to CSV.jl)
https://github.com/JuliaStats/DataFrames.jl/issues/304
https://github.com/JuliaStats/DataFrames.jl/issues/292
https://github.com/JuliaStats/DataFrames.jl/issues/291
https://github.com/JuliaStats/DataFrames.jl/issues/495
https://github.com/JuliaStats/DataFrames.jl/issues/430
https://github.com/JuliaStats/DataFrames.jl/issues/425
https://github.com/JuliaStats/DataFrames.jl/issues/402
https://github.com/JuliaStats/DataFrames.jl/issues/399
https://github.com/JuliaStats/DataFrames.jl/issues/385
https://github.com/JuliaStats/DataFrames.jl/issues/770
https://github.com/JuliaStats/DataFrames.jl/issues/736
https://github.com/JuliaStats/DataFrames.jl/issues/678
https://github.com/JuliaStats/DataFrames.jl/issues/672
https://github.com/JuliaStats/DataFrames.jl/issues/616
https://github.com/JuliaStats/DataFrames.jl/issues/608
https://github.com/JuliaStats/DataFrames.jl/issues/607
https://github.com/JuliaStats/DataFrames.jl/issues/587
https://github.com/JuliaStats/DataFrames.jl/issues/586
https://github.com/JuliaStats/DataFrames.jl/issues/584
https://github.com/JuliaStats/DataFrames.jl/issues/568
https://github.com/JuliaStats/DataFrames.jl/issues/551
https://github.com/JuliaStats/DataFrames.jl/issues/530
https://github.com/JuliaStats/DataFrames.jl/issues/942
https://github.com/JuliaStats/DataFrames.jl/issues/924
https://github.com/JuliaStats/DataFrames.jl/issues/878
https://github.com/JuliaStats/DataFrames.jl/issues/813
https://github.com/JuliaStats/DataFrames.jl/issues/800
https://github.com/JuliaStats/DataFrames.jl/issues/1213
https://github.com/JuliaStats/DataFrames.jl/issues/1211
https://github.com/JuliaStats/DataFrames.jl/issues/1188
https://github.com/JuliaStats/DataFrames.jl/issues/1080
https://github.com/JuliaStats/DataFrames.jl/issues/1044
https://github.com/JuliaStats/DataFrames.jl/issues/1011
https://github.com/JuliaStats/DataFrames.jl/issues/986
other
https://github.com/JuliaStats/DataFrames.jl/issues/485 fixed by https://github.com/JuliaData/DataTables.jl/pull/28
https://github.com/JuliaStats/DataFrames.jl/issues/602 fixed by https://github.com/JuliaData/DataTables.jl/pull/6
https://github.com/JuliaStats/DataFrames.jl/issues/973 fixed by https://github.com/JuliaData/DataTables.jl/pull/64
https://github.com/JuliaStats/DataFrames.jl/issues/900 (I think this just works)
https://github.com/JuliaStats/DataFrames.jl/issues/1119 (will be) fixed by https://github.com/JuliaData/DataTables.jl/pull/64 & https://github.com/JuliaData/DataTables.jl/pull/66
https://github.com/JuliaStats/DataFrames.jl/issues/1091 fixed by https://github.com/JuliaData/DataTables.jl/pull/64
https://github.com/JuliaStats/DataFrames.jl/issues/985 fixed by https://github.com/JuliaData/DataTables.jl/pull/17 and https://github.com/JuliaData/DataTables.jl/pull/79

Current failure on nightly appears to be from CSV.jl. Maybe we should switch to allow failures on nightly while we get all the related packages updated?

That seems totally fine to me. I have a whole load of updates coming next week for DataStreams + CSV for 0.6/nightly.

Thanks. I think we should apply all fixes you added to DataTables first (except of course those about renaming to DataFrame), and then backport them just like other commits. Once https://github.com/JuliaData/DataTables.jl/pull/66 will have been merged, we'll also backport it to this PR, and only then we should merge it: that will avoid having a version of DataFrames master based on Nullable only for a few weeks.

Sorry, you're wanting to apply the fixes back into DataTables.jl? I don't think that would go very smoothly since they're largely based on commits in DataFrames.jl that don't existing in DataTables.jl. I'm fine with waiting for #66 to get merge and apply those changes to this PR though.

AFAIK this branch is basically DataTables without the renaming, so most changes should apply fine, right? By "fixes you added", I just meant the seven commits which have no equivalent in DataTables yet (not those I rebased in my original branch).
Regarding https://github.com/JuliaData/DataTables.jl/pull/66, I've realized we should probably not merge it in DataTables at all, but just include it in this PR once it is ready.

By "fixes you added", I just meant the seven commits which have no equivalent in DataTables yet (not those I rebased in my original branch).

Yes, I believe I always chose to keep the new DataTables code when there were conflicts during the rebase with the DataFrames master branch. It just takes a while when making notes of what that breaks during the rebase.

Regarding JuliaData/DataTables.jl#66, I've realized we should probably not merge it in DataTables at all, but just include it in this PR once it is ready.

Yeah, my understanding was that we'd probably just apply those changes to this branch between the last of the backport commits and the cleanup ones I added (that should be squashed).

Yes, I believe I always chose to keep the new DataTables code when there were conflicts during the rebase with the DataFrames master branch. It just takes a while when making notes of what that breaks during the rebase.

What did you fo exactly? The point of my branch was that there were no conflicts at all because I just took the DataTables history and used an automated script to remove all traces of the renaming to DataTable. It was not supposed to be rebased on the DataFrames history since the fork. I'd rather not apply any other changes to the history so that we can easily follow it (and its correspondence with PRs). There should be very few changes to backport from DataFrames that are not in DataTables.

I checked out nl/datatables2 and ran git pull --rebase origin master to apply the DataTables.jl commits onto the current master commits (the same thing I'd do for any other feature branch). As it applied the commits from your branch there were conflicts (e.g., different changes to printtable in both DataFrames and DataTables). For these conflicts, I accepted the DataTable changes, but made a note of what was being overwritten, so that I could fix them up after completing the rebase.

I see. I'd rather not do that, since most commits in DataFrames since the fork are either already in DataTables or are not needed there. Merging these two histories is going to create confusion without any clear advantages. It will be clearer to throw away everything that happened in DataFrames since the fork and only backport the (very few) needed commits manually.

It will be clearer to throw away everything that happened in DataFrames since the fork and only backport the (very few) needed commits manually.

So, essentially rewrite the history on the DataFrames master branch for the past year? That sounds like a really bad idea... I agree that rebasing and merging in changes from DataTables will result in a slightly uglier git history, but in all fairness what we're wanting to do is kind of ugly and that should probably be preserved in the git history. As it stands, I really only see 2 reasonable options.

Merge the DataTables changes into DataFrames like any other feature branch and accept that it'll have slightly ugly history or
Take the commits we want from DataFrames and create a feature branch to DataTables.

I don't think that we should rewrite the history on the master branch in either project though.

+1 for option 1

I personally like the term Table over DataFrame, but I'm not really bothered either way.

Honestly the best solution would have been not to create DataTables in the first place, and keep all the development in DataFrames. Or to freeze DataFrames now and continue with DataTables. But when discussed that a few weeks ago people preferred keeping DataFrames because of the user base and because the name is more familiar.
Regarding the way to handle history, I really don't think rebasing DataTables on DataFrames makes sense. Yes, it sounds more correct at first, but that's really just an illusion. DataTables is not a feature branch to DataFrames. It has lived in parallel and has accumulated many more changes than DataFrames (even leaving aside the renaming). There have been 80 commits in DataTables since the fork (including major refactoring in several areas), versus only 20 in DataFrames (mostly small changes or changes related to DataArrays). I systematically ensured that useful fixes to DataFrames were also applied to DataTables. So rebasing really implies losing the DataTables history by rewriting commits manually on top of DataFrames, which is arguably worse than losing the DataFrames history since the fork.
What I suggest is not to lose all the DataFrames git history, it's to move it to a release-0.10 branch; that way the old releases can still be installed transparently. The state of master would then clearly reflect the fact that all the commits from DataTables were not written taking into account the state of the DataFrames codebase at that time.
IMHO the only other sane option is to perform a plain merge (not rebase) with a big merge commit fixing the conflicts. That would at least preserve the history of the DataTables commits without any manual changes inside them. That's not pretty either, but that's the most honest accounting of the development history. I'm not convinced it's more useful than creating a clean branch from DataTables, but if you prefer keeping all of the DataFrames history in master...

There have been 80 commits in DataTables since the fork (including major refactoring in several areas), versus only 20 in DataFrames (mostly small changes or changes related to DataArrays).

Hmmm, that does support the idea of deprecating DataFrames and moving forward with DataTables, but I guess this has already be decided.

IMHO the only other sane option is to perform a plain merge (not rebase) with a big merge commit fixing the conflicts. That would at least preserve the history of the DataTables commits without any manual changes inside them. That's not pretty either, but that's the most honest accounting of the development history. I'm not convinced it's more useful than creating a clean branch from DataTables, but if you prefer keeping all of the DataFrames history in master...

I'd feel more comfortable rebasing as it makes it easier to keep track of conflicts rather than 1 giant merge. I suppose a middle ground might be to maintain a DataFrames branch of the DataTables code we want to merge in (that gets rebased regularly) and then we use that final state of that branch for handling conflicts in the final merge. This would preserve history because we'd just be merging in the original commits (vs the rebased commits), but it'll be easier and less error prone for handling conflicts.

I'd feel more comfortable rebasing as it makes it easier to keep track of conflicts rather than 1 giant merge.

The problem is that rebasing does not allow tracking the conflicts: it just hides them by modifying each commit. Original commits are no longer available. It's also very easy to end up with individual commits which make no sense.

I suppose a middle ground might be to maintain a DataFrames branch of the DataTables code we want to merge in (that gets rebased regularly) and then we use that final state of that branch for handling conflicts in the final merge. This would preserve history because we'd just be merging in the original commits (vs the rebased commits), but it'll be easier and less error prone for handling conflicts.

Could you elaborate? I don't understand this suggestion.

The problem is that rebasing does not allow tracking the conflicts: it just hides them by modifying each commit. Original commits are no longer available.

That's true, so if we want to preserve the commits on that branch then we'll want to merge and deal with the conflicts all at once. My point was more that, as a person who isn't particularly familiar with this code base, addressing conflicts during a large rebase helps with systematically addressing conflicts as they pop up (or at least making a note to fix it afterwards). My issue with large merges is that it's easy to miss something and have it go unnoticed.

Could you elaborate? I don't understand this suggestion.

I was just suggesting that we use a rebased branch as a template of what we want the merged state to look like. Ideally, rebasing where we always keep the changes on the branch and apply fixes afterwards should leave the codebase in the same state as a large merge. In theory, we could even choose to resolve conflicts in the merge by checking out files from the rebased branch.

My issue with large merges is that it's easy to miss something and have it go unnoticed.

Yes, but it's also quite hard to rebase 80 commits and ensure none of them gets broken in some way (i.e. that all of them pass the tests and make sense separately, including that their commit messages still describe the changes they apply). But indeed as you suggest we can apply both approaches and check that they give the same result. We just need to make a final decision about which history we keep.

My issue with large merges is that it's easy to miss something and have it go unnoticed.

Yes, but it's also quite hard to rebase 80 commits and ensure none of them gets broken in some way (i.e. that all of them pass the tests and make sense separately, including that their commit messages still describe the changes they apply). But indeed as you suggest we can apply both approaches and check that they give the same result. We just need to make a final decision about which history we keep.

To be honest at this point I don't really care how we do the merging/rebasing/whatever, as long as we get the relevant DataTables stuff into here and authorship information is retained. I could be missing something or oversimplifying but it doesn't seem like it warrants a big, in-depth discussion; just do whatever works. ¯\_(ツ)_/¯ IMO anyway.

Yes, the way we backport the changes is clearly secondary compared with the goal of getting it done, but better do it correctly since any mistake here will be painful for a long time. Anyway we're mostly waiting on JuliaData/DataTables.jl#66.
For reference and reproducibility, here is the script I've used with git filter-branch to erase all traces of the rename from DataFrame to DataTable from the history. I'll try to update it to also fix remaining problems like DTRowIterator.

Turns out it was easy. Could you have a look at the nl/datatables3 and check that it fixes the naming issues you spotted?

Yep, looks like all the DT* cases have been fixed, thanks. We'll still have a bunch of conflicts when merging with master, but at least we won't have reference to DataTables and DTs in the history :)

I couldn't find any issues with nl/datatables3 either, looks great!

Cool. What's the way forward? Would you prepare a merge commit of nl/datatables3 into master? The contents of the files should be identical to what you obtained by rebasing AFAICT. Then we should be able to apply your additional fixes and JuliaData/DataTables.jl#66 on top of that. When we're ready we'll just have to rename master to release-0.10 and replace it with that new branch.

I'll probably keep rebasing in changes for this branch (e.g., JuliaData/DataTables.jl#66), so we can review changes as the come in. When we're comfortable with all the backported changes then I'll create a release-0.12 branch off master (we'll probably want to tag a 0.11.0 release first) and merge nl/datatables3 into that branch.

I think we can tag 0.10.1 rather than 0.11, couldn't we? Changes are not that big.

If we want #1209 in before the backport then that would probably warrant a minor release.

Depends on whether those changes are user-facing. The changes in DataArrays are, but not necessarily the adaptations made in DataFrames. I'll have to look closer at that.

What's the best way to pull https://github.com/JuliaData/DataTables.jl/pull/66 over here? I don't quite understand the git situation here; some kind of cross-repo cherry-pick? Or just copy/paste the files over and rename?

As discussed above, my preferred option would be to do git checkout master && git merge nl/datatables3 to preserve both histories, with a merge commit to fix conflicts (which should be reasonably big). JuliaData/DataTables.jl#66 commits can be added either to nl/datatables3 before merging, or to master after merging. Maybe before merging is slightly better so that Nullable isn't used in the code base after the merge commit, which is more consistent with the fact that DataFrames never used it.

I'd go with @nalimilan's suggestion, but I'd suggest making a release branch off master first and merging into that so we can review the merged state before it goes to master. I might also pull those commits into this branch as well so we can diff the two branches.

Rebased. Porting over #66 now.

@quinnj Also push the commits to nl/datatables3 when they are ready, that will be the branch to use in the end.

Ok, got a good start here. Merged #66 with this branch and resolved all the conflicts/renaming issues (there are still a few dt-like variable names hanging around). Now it's just getting all tests passing.

I've included the new changes into nl/datatables4, and merged it with a merge commit into a new nl/datatables-backport branch which was equivalent to master: https://github.com/JuliaStats/DataFrames.jl/pull/1220. If that's OK, I can do the same directly on master (or rename master to release-0.9 and nl/datatables-backport to master).

I'll note that I was planning on rebasing (vs merging) the changes from DataTables#66 into this branch. The rf/datatables and nl/datatables-backport branches are very different now, so I'm just going to close this.


Restore AKS Base Ubuntu March 2020 version, and update support matrix link anchor

Per @walterov : Could we address this as an URGENT issue? It will generate support calls.
Version: AKS Base Ubuntu 16.04-LTS Image Distro, March 2020 (2020.03.19)
Latest AKS Engine 0.48.0 only matches the version above: https://docs.microsoft.com/en-us/azure-stack/user/kubernetes-aks-engine-release-notes?view=azs-2002#download-new-image-and-aks-engine
Mapping of image and aks-engine versions: https://github.com/Azure/aks-engine/blob/master/docs/topics/azure-stack.md#supported-aks-engine-versions
(Currently used anchor #supported-kubernetes-versions is broken- https://github.com/Azure/aks-engine/blob/master/docs/topics/azure-stack.md#supported-kubernetes-versions)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bcfb60cd-c79d-e4c3-9015-bb3b49e1dc06
Version Independent ID: 9a48b797-af3b-462d-94a0-07693651a488
Content: Add the Azure Kubernetes Services (AKS) engine prerequisites to the Azure Stack Hub Marketplace - Azure Stack Hub
Content Source: azure-stack/operator/azure-stack-aks-engine.md
Service: azure-stack
GitHub Login: @mattbriggs
Microsoft Alias: mabrigg

For the March 2020 base image, looks like it was added on April 24- https://github.com/MicrosoftDocs/azure-stack-docs-pr/pull/3444/files#diff-562d1057d338ff57057c1ca099b43150R57
But then reverted on April 27, when the image was temporarily rolled back-
https://github.com/MicrosoftDocs/azure-stack-docs/commit/22a8091495898ca9701b9665a8e0c494f1a188d7#diff-562d1057d338ff57057c1ca099b43150L57

#in-progress

@alexsmithMSFT @walterov Thanks for passing along this info. I've updated the links that had the old anchor and the operator topic with the right version number: https://review.docs.microsoft.com/en-us/azure-stack/operator/azure-stack-aks-engine?view=azs-1802&branch=pr-en-us-3864 You can find the pr here: https://github.com/MicrosoftDocs/azure-stack-docs-pr/pull/3864

#please-close


Looking for new maintainer(s)

@rohitjha  @gkampolis @anovik @agdenadel
Hi everyone,
Apologies for having not responded to your PRs and interest in the project.
Because of priorities at Stratajet and this not being a heavily used package for us, I've not been able to maintain it like I maintain some of my other OSS packages. I would love for UnitConversion to continue to exist if you're all getting value in it, but just can't commit the time myself.
This in mind, would anyone be interested in taking over as maintainer? Either reply here or throw me an email if so, and we can talk about a handover of things like the nuget repo.
Nick

I'm going to have some time on my hands for the second half of December and after New Year's, so maybe we could arrange something (I was planning to add density units anyway and who knows what else).
However this would be the first time I'd become a maintainer for an existing project, so some guidance on the procedures involved would be greatly appreciated!
Kind regards,
George

Great! I'm happy to walk you through some bits.
It's all quite straight-forward though. You just need to set up a free AppVeyor account and create a project for your fork of the repo. Most of the configuration is in the repo, so it's just some security stuff like the key for deploying to Nuget.

Hi @Nick-Lucas  @gkampolis
May I ask you about the status of other PRs made to Stratajet? Will they be merged to new repo?
Thanks.

@anovik
I'm going through the process of writing a contribution.md and some style cop basic rules to settle on variable naming (PascalCase etc.), everything should be done sometime next week, depending on available time.
You should be making all pull requests "against" my repo of the project, here.

@anovik If you're so inclined, I've set up a gitter chatroom. We can talk further about moving your pull requests over there.
Of course, everyone is more than welcome to join in and participate.

@Nick-Lucas hi there, I've pretty much set up everything up and all is going well.
However I noticed that my is still listed as a fork and not the "home" of the project.

I thought that you may want to transfer this repo to me, this would solve the problem posited by @anovik as the pull requests to Stratajet's repo would be now handled by me, ensuring continuity instead of forcing everyone to migrate to the new location.
If you agree to this, I'll create a new pull request with the changes I've doe so far and then you can transfer this repo to me (in the repo's settings).
Your thoughts?

Looks like I can't directly transfer it to you, but can make you an admin for it.
With Admin rights you can likely resolve this, as I believe you can transfer out of the organisation to your own account.
Let me know if I can do anything else

@Nick-Lucas That worked! Thanks for being so prompt! 👍


Source code links are broken in the documentation

I noticed this issue on this page.
https://googlecloudplatform.github.io/gcloud-python/#/docs/master/gcloud/bigquery/client/client
For example, the source code link for the init method links to https://github.com/GoogleCloudPlatform/gcloud-python/blob/master//home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/json-docs/lib/python2.7/site-packages/gcloud/client.py/gcloud/client.py#L184 which 404s.

There is a PR to fix this that hasn't been merged yet. I'm out of the country at the moment with spotty wifi. I can look at it tonight possibly though.

Closing this as a duplicate of https://github.com/GoogleCloudPlatform/gcloud-python/issues/2046.


Is pow retrying requests if no response is received

I'm using pow to proxy requests to my local Tomcat server (which works fabulously btw). However, I ran into a weird issue recently where I was watching a long-running request in my Tomcat debugger and after a while an identical request came in on a different Tomcat thread. I verified in the browser's network activity that it was not retrying the request. Then I removed pow from the equation and the problem went away.
Is pow retrying requests if it doesn't receive a response after a period of time? Is there a way to disable that functionality or increase the timeout?
Thanks in advance!!

@bradical fancy seeing you here.
I'm pretty sure this is indeed a pow / nack issue... I just ran into the same thing with:
browser -> pow -> Rails app
Setting POW_WORKERS env var to 1 fixed it. ¯_(ツ)_/¯

Nice. Thanks @etipton!