Support skipping tar entries

It should be possible to efficiently "skip" tar entries we're not interested in. For instance, consider something like
final reader = tar.Reader(file.openRead());
tar.Entry entry;
while ((entry = await reader.next() != null) {
  if (entry.typeFlag != tar.TypeFlag.reg || !myGlob.matches(entry.path)) continue;

  // read entry
}

I think the reader can just cancel the current substream when next gets called and the active stream hasn't been listened to.

Currently, if you forget to drain the entry, like:
void main() async {
  final tarStream = File('archive.tar').openRead().transform(reader);
  await for (final entry in tarStream) {
    await Future.delayed(Duration(milliseconds: 1));
    print(entry.name);
    // entry.drain(); // without this we only get the first file...
  }
}

Then reading of the remaining entries halts.

I think this is why we want away from having a Stream<Entry> interface in the implementation we did with dart-neats.
Because if the user forgets to read the entries substream for an entry, then the reader completely halts. There could be an exception, a parsing error or something else that causes an entry to be only partially read.
Moreover, there is no way that the stream producer can throw an error or drain the substream, if the user forgets to drain the substream. Thus, it might be safest to move away from the StreamTransformer interface, as this is a pretty easy footgun for users to trigger :foot::gun: (and it's kind of hard to debug, especially when we can't even throw an error telling users what they did wrong).

I would suggest an interface like:
abstract class TarReader {
  factory TarReader(Stream<List<int>> tarStream) => ...
  /// Read next entry, false if we are EOF.
  ///
  /// Throws [StateError] if the [contents] from the previous entry have not been drained.
  Future<bool> next();

  /// Header for current entry, null if we are EOF, the call the next() haven't returned yet.
  Header header;

  /// Contents for current entry, null if we are EOF, the call the next() haven't returned yet.
  Stream<List<int>> contents;

  /// Utility function for quickly iterating through all entries in [tarStream].
  static Future<void> forEach(
    Stream<List<int>> tarStream,
    FutureOr<void> Function(Header header, Stream<List<int>> contents) action
  ) async {
    final reader = TarReader(tarStream);
    while(await reader.next()) {
      await action(reader.header, reader.contents);
    }
  }
}

Calling reader.next() is not as elegant, but it's simple and most people can figure it out --- and most importantly we can throw an error if they don't fully drain contents from the current entry before trying to read the next.
To make it even safer to use we could wrap the stream we return in contents with a StreamController such that we can detect if the user has started subscribing to it; and or fully consumed it.
This way, when Reader.next() is called we have the following cases:

contents is fully drained or cancelled, we continue reading the next entry.
Listen has never been called on contents, in this case it's probably safe to drain on behalf of the user.
contents have been listened to, but not fully consumed, we throw a StateError.

We could wait for contents to be fully consumed, but then if the user forgot we would just deadlock and not return the remaining entries from the tarStream -- which is the footgun we're trying to avoid building :D
We could also choose to drain contents on behalf of the user... But if the user is writing it to a file and forgot to await this operation, then the user will end up writing partial files in some cases (probably someone will experience that small files are successfully written, but large files are truncated, because this will be affected by events and timing).
Throwing a StateError if not fully consumed is a bit crude, but at least it gives fairly consistent behavior.

Another upside with an interface like this is that there is no need for an Entry class. The writer can just take Writer.write(Header, Stream<List<int>>), and this might be better because you kind of want to encourage the user to do await writer.write(header, File(...).openRead()) otherwise, the user will have multiple file streams open at once (not that this is a big deal).

I did a bit of digging and it seems streams in Dart are not required to suspend the same way asynchronous generators are in Javascript. Don't quote me on this, but try to run the following in on the Dart VM:
void main() async {
  final stream = () async* {
    print('Creating Object 1:');
    yield 'Object 1';

    // This is printed before "GOT: Object 1", is it weird that this surprises me?
    print('Creating Object 2:');
    yield 'Object 2';

    print('Creating Object 3:');
    yield 'Object 3';

    print('done');
  }();

  await for (final m in stream) {
    await Future.delayed(Duration(milliseconds: 1));
    print('Consumed: $m');
  }
}

You'll notice that "Creating Object 2", will be printed before "Consumed: Object 1" is printed.
I guess it makes sense that a stream of events should start producing the next event as soon as the first event has been accepted.

Currently, if you forget to drain the entry, like:
void main() async {
  final tarStream = File('archive.tar').openRead().transform(reader);
  await for (final entry in tarStream) {
    await Future.delayed(Duration(milliseconds: 1));
    print(entry.name);
    // entry.drain(); // without this we only get the first file...
  }
}

Then reading of the remaining entries halts.

I think this is why we want away from having a Stream<Entry> interface in the implementation we did with dart-neats.
Because if the user forgets to read the entries substream for an entry, then the reader completely halts. There could be an exception, a parsing error or something else that causes an entry to be only partially read.
Moreover, there is no way that the stream producer can throw an error or drain the substream, if the user forgets to drain the substream. Thus, it might be safest to move away from the StreamTransformer interface, as this is a pretty easy footgun for users to trigger :foot::gun: (and it's kind of hard to debug, especially when we can't even throw an error telling users what they did wrong).

I would suggest an interface like:
abstract class TarReader {
  factory TarReader(Stream<List<int>> tarStream) => ...
  /// Read next entry, false if we are EOF.
  ///
  /// Throws [StateError] if the [contents] from the previous entry have not been drained.
  Future<bool> next();

  /// Header for current entry, null if we are EOF, the call the next() haven't returned yet.
  Header header;

  /// Contents for current entry, null if we are EOF, the call the next() haven't returned yet.
  Stream<List<int>> contents;

  /// Utility function for quickly iterating through all entries in [tarStream].
  static Future<void> forEach(
    Stream<List<int>> tarStream,
    FutureOr<void> Function(Header header, Stream<List<int>> contents) action
  ) async {
    final reader = TarReader(tarStream);
    while(await reader.next()) {
      await action(reader.header, reader.contents);
    }
  }
}

Calling reader.next() is not as elegant, but it's simple and most people can figure it out --- and most importantly we can throw an error if they don't fully drain contents from the current entry before trying to read the next.
To make it even safer to use we could wrap the stream we return in contents with a StreamController such that we can detect if the user has started subscribing to it; and or fully consumed it.
This way, when Reader.next() is called we have the following cases:

contents is fully drained or cancelled, we continue reading the next entry.
Listen has never been called on contents, in this case it's probably safe to drain on behalf of the user.
contents have been listened to, but not fully consumed, we throw a StateError.

We could wait for contents to be fully consumed, but then if the user forgot we would just deadlock and not return the remaining entries from the tarStream -- which is the footgun we're trying to avoid building :D
We could also choose to drain contents on behalf of the user... But if the user is writing it to a file and forgot to await this operation, then the user will end up writing partial files in some cases (probably someone will experience that small files are successfully written, but large files are truncated, because this will be affected by events and timing).
Throwing a StateError if not fully consumed is a bit crude, but at least it gives fairly consistent behavior.

Another upside with an interface like this is that there is no need for an Entry class. The writer can just take Writer.write(Header, Stream<List<int>>), and this might be better because you kind of want to encourage the user to do await writer.write(header, File(...).openRead()) otherwise, the user will have multiple file streams open at once (not that this is a big deal).

I did a bit of digging and it seems streams in Dart are not required to suspend the same way asynchronous generators are in Javascript. Don't quote me on this, but try to run the following in on the Dart VM:
void main() async {
  final stream = () async* {
    print('Creating Object 1:');
    yield 'Object 1';

    // This is printed before "GOT: Object 1", is it weird that this surprises me?
    print('Creating Object 2:');
    yield 'Object 2';

    print('Creating Object 3:');
    yield 'Object 3';

    print('done');
  }();

  await for (final m in stream) {
    await Future.delayed(Duration(milliseconds: 1));
    print('Consumed: $m');
  }
}

You'll notice that "Creating Object 2", will be printed before "Consumed: Object 1" is printed.
I guess it makes sense that a stream of events should start producing the next event as soon as the first event has been accepted.

Another upside to doing the TarReader interface is that we can use ChunkedStreamIterator and have it be something like:
import 'package:pedantic/pedantic.dart' show unawaited;
import 'package:chunked_stream/chunked_stream.dart' show ChunkedStreamIterator;

class TarReader {
  final ChunkedStreamIterator<int> _input;

  TarReader(Stream<List<int>> tarStream)
      : _input = ChunkedStreamIterator(tarStream);

  Header? header;
  Stream<List<int>>? contents;

  Future<bool> next() async {
    // Prevent anyone from reading header while we're parsing
    header = null;
    contents = null;

    var block = await _input.readBlock();
    if (block.length < 512) {
      // We are EOF
    }
    if (block.isAllZeroes) {
      block = await _input.readBlock();
      if (block.isAllZeroes) {
        try {
          // there can be more blocks, if two tar files were concatenated
          block = await _input.readBlock();
        } on UnexpectedEOFException {
          return false; // here EOF is actually expected
        }
      }
    }

    final h = HeaderImpl.parseBlock(block);
    // TODO: read all the various PAX headers and GNU extensions, etc...

    header = h;
    contents = _input.substream(header.size);
    return true;
  }
}

extension on ChunkedStreamIterator<int> {
  Future<List<int>> readBlock() async {
    final block = await read(512):
    if (block.length < 512) {
      throw UnexpectedEOFException();
    }
    return block;
  }
}

Using .read(512) and .substream(header.size) from ChunkedStreamIterator will ideally make the code easier to read too :D
There will be no need to keep track of state outside of the TarReader.next() function. No need to propagate pause/resume. No need to manage buffers because the next chunk wasn't a fill 512 bytes. And no need to track if the user fully consumed TarReader.contents because ChunkedStreamIterator will throw  if _input.substream returned a non-empty substream that wasn't fully consumed, canceled or drained.

Another upside to doing the TarReader interface is that we can use ChunkedStreamIterator and have it be something like:
import 'package:pedantic/pedantic.dart' show unawaited;
import 'package:chunked_stream/chunked_stream.dart' show ChunkedStreamIterator;

class TarReader {
  final ChunkedStreamIterator<int> _input;

  TarReader(Stream<List<int>> tarStream)
      : _input = ChunkedStreamIterator(tarStream);

  Header? header;
  Stream<List<int>>? contents;

  Future<bool> next() async {
    // Prevent anyone from reading header while we're parsing
    header = null;
    contents = null;

    var block = await _input.readBlock();
    if (block.length < 512) {
      // We are EOF
    }
    if (block.isAllZeroes) {
      block = await _input.readBlock();
      if (block.isAllZeroes) {
        try {
          // there can be more blocks, if two tar files were concatenated
          block = await _input.readBlock();
        } on UnexpectedEOFException {
          return false; // here EOF is actually expected
        }
      }
    }

    final h = HeaderImpl.parseBlock(block);
    // TODO: read all the various PAX headers and GNU extensions, etc...

    header = h;
    contents = _input.substream(header.size);
    return true;
  }
}

extension on ChunkedStreamIterator<int> {
  Future<List<int>> readBlock() async {
    final block = await read(512):
    if (block.length < 512) {
      throw UnexpectedEOFException();
    }
    return block;
  }
}

Using .read(512) and .substream(header.size) from ChunkedStreamIterator will ideally make the code easier to read too :D
There will be no need to keep track of state outside of the TarReader.next() function. No need to propagate pause/resume. No need to manage buffers because the next chunk wasn't a fill 512 bytes. And no need to track if the user fully consumed TarReader.contents because ChunkedStreamIterator will throw  if _input.substream returned a non-empty substream that wasn't fully consumed, canceled or drained.

Thanks for all the suggestions! I thought about it, and agree that emitting entries as a stream is just too easy to misuse. I'll adapt the reader to use the interface you suggested.

To make it even safer to use we could wrap the stream we return in contents with a StreamController such that we can detect if the user has started subscribing to it; and or fully consumed it.

Agreed, I think we should handle the three cases you described. I need to wrap the internal stream either way, to guard against an unexpected EoF where substream would return fewer bytes than indicated in the header.

I did a bit of digging and it seems streams in Dart are not required to suspend the same way asynchronous generators are in Javascript. Don't quote me on this, but try to run the following in on the Dart VM:

I think that's a bug in the VM though, it works on dartpad. https://dartbug.com/34775

Thanks for all the suggestions! I thought about it, and agree that emitting entries as a stream is just too easy to misuse. I'll adapt the reader to use the interface you suggested.

To make it even safer to use we could wrap the stream we return in contents with a StreamController such that we can detect if the user has started subscribing to it; and or fully consumed it.

Agreed, I think we should handle the three cases you described. I need to wrap the internal stream either way, to guard against an unexpected EoF where substream would return fewer bytes than indicated in the header.

I did a bit of digging and it seems streams in Dart are not required to suspend the same way asynchronous generators are in Javascript. Don't quote me on this, but try to run the following in on the Dart VM:

I think that's a bug in the VM though, it works on dartpad. https://dartbug.com/34775

I think that's a bug in the VM though, it works on dartpad. https://dartbug.com/34775

Nice, find!
Reading the spec 17.6.3 (page 152), it's also my understanding that any await inside the body of await for should pause the stream being consumed:

Either execution of s is completely synchronous, or it contains an asynchronous construct (await, await for, yield or yield*) which will pause the stream subscription of its surrounding asynchronous loop.

I think that's a bug in the VM though, it works on dartpad. https://dartbug.com/34775

Nice, find!
Reading the spec 17.6.3 (page 152), it's also my understanding that any await inside the body of await for should pause the stream being consumed:

Either execution of s is completely synchronous, or it contains an asynchronous construct (await, await for, yield or yield*) which will pause the stream subscription of its surrounding asynchronous loop.

Thinking more about this, if you have an instance of Stream<Entry> and you must drain Entry.stream before reading the next entry, then all of the utility method on Stream<T> are pretty much useless.
final entryStream = File('archive.tar').openRead().transform(tarReader);

// None of this methods on Stream<T> can work because they won't drain the substreams
await entryStream.length;
await entryStream.last;
await entryStream.any(...);
await entryStream.forEach(...);
await entryStream.contains(...);
await entryStream.skipWhile(...);
// etc...

The only sensible things you can do with an instance of Stream<Entry> is to use the await for syntax, or call entryStream.asyncMap(...).
So returning an instance of Stream<Entry> is perhaps not very sensible, because you use it in most of the ways a normal Stream<T> can be used. You can't do something like:
// This won't work
final hasPubspec = File('archive.tar').openRead().transform(tarReader).any(
  (entry) => entry.header.name == 'pubspec.yaml'
);

Just sharing my notes here...

@lrhn suggest we make TarReader a subclass of StreamIterator which already has a moveNext() and a current property.
So something like:

class TarReader extends StreamIterator<Header> {
  @override
  Header get current;

  Stream<List<int>> get contents;

  @override
  Future<bool> moveNext() async {...}

  @override
  Future<void> cancel() async {...}
}

Thinking more about this, if you have an instance of Stream<Entry> and you must drain Entry.stream before reading the next entry, then all of the utility method on Stream<T> are pretty much useless.
final entryStream = File('archive.tar').openRead().transform(tarReader);

// None of this methods on Stream<T> can work because they won't drain the substreams
await entryStream.length;
await entryStream.last;
await entryStream.any(...);
await entryStream.forEach(...);
await entryStream.contains(...);
await entryStream.skipWhile(...);
// etc...

The only sensible things you can do with an instance of Stream<Entry> is to use the await for syntax, or call entryStream.asyncMap(...).
So returning an instance of Stream<Entry> is perhaps not very sensible, because you use it in most of the ways a normal Stream<T> can be used. You can't do something like:
// This won't work
final hasPubspec = File('archive.tar').openRead().transform(tarReader).any(
  (entry) => entry.header.name == 'pubspec.yaml'
);

Just sharing my notes here...

@lrhn suggest we make TarReader a subclass of StreamIterator which already has a moveNext() and a current property.
So something like:

class TarReader extends StreamIterator<Header> {
  @override
  Header get current;

  Stream<List<int>> get contents;

  @override
  Future<bool> moveNext() async {...}

  @override
  Future<void> cancel() async {...}
}

Ah, TIL about StreamIterator!

then all of the utility method on Stream<T> are pretty much useless.

Well, we implicitly drain the content stream if it's never listened to, so they should work in general. The hard part is deciding how long to wait before emitting another value since we're essentially a Stream<Stream<List<int>>>. When emitting an entry, we can't really know if anyone will ever listen to it or if we're supposed to drain it.
In my old tar implementation, I always emitted data through the entry stream so it doesn't make a difference if one listens to the stream or not. That supports skipping entries, but you might miss parts of a file if you wait before listening (which is even worse). Oh well, StreamIterator it is.

Ah, TIL about StreamIterator!

then all of the utility method on Stream<T> are pretty much useless.

Well, we implicitly drain the content stream if it's never listened to, so they should work in general. The hard part is deciding how long to wait before emitting another value since we're essentially a Stream<Stream<List<int>>>. When emitting an entry, we can't really know if anyone will ever listen to it or if we're supposed to drain it.
In my old tar implementation, I always emitted data through the entry stream so it doesn't make a difference if one listens to the stream or not. That supports skipping entries, but you might miss parts of a file if you wait before listening (which is even worse). Oh well, StreamIterator it is.

Well, we implicitly drain the content stream if it's never listened to, so they should work in general. The hard part is deciding how long to wait before emitting another value since we're essentially a Stream<Stream<List<int>>>.

That's true, then if https://dartbug.com/34775 was fixed it might work reasonably.
Anyways, let's not think too much about the interface, if it's reasonably simple and foolproof, we can always make it elegant later :D

I always emitted data through the entry stream so it doesn't make a difference if one listens to the stream or not. That supports skipping entries, but you might miss parts of a file if you wait before listening (which is even worse).

So you made the entry stream into a Broadcast stream, otherwise you can't force the events out... Or well, if you do they become buffered in memory. I guess either scenario is kind of undesirable.
In any case, I'm guessing StreamIterator with moveNext will work well with ChunkedStreamIterator behind the scenes. If we need/want any modifications to ChunkedStreamIterator let me know. Maybe we should tweak it be ChunkedStreamIterator<T extends List> such that we can get ChunkedStreamIterator<Uint8List> -- or maybe we just need Uint8ListStreamIterator so as to not get too fancy with generics :)

Well, we implicitly drain the content stream if it's never listened to, so they should work in general. The hard part is deciding how long to wait before emitting another value since we're essentially a Stream<Stream<List<int>>>.

That's true, then if https://dartbug.com/34775 was fixed it might work reasonably.
Anyways, let's not think too much about the interface, if it's reasonably simple and foolproof, we can always make it elegant later :D

I always emitted data through the entry stream so it doesn't make a difference if one listens to the stream or not. That supports skipping entries, but you might miss parts of a file if you wait before listening (which is even worse).

So you made the entry stream into a Broadcast stream, otherwise you can't force the events out... Or well, if you do they become buffered in memory. I guess either scenario is kind of undesirable.
In any case, I'm guessing StreamIterator with moveNext will work well with ChunkedStreamIterator behind the scenes. If we need/want any modifications to ChunkedStreamIterator let me know. Maybe we should tweak it be ChunkedStreamIterator<T extends List> such that we can get ChunkedStreamIterator<Uint8List> -- or maybe we just need Uint8ListStreamIterator so as to not get too fancy with generics :)

If we need/want any modifications to ChunkedStreamIterator let me know

For performance, I think it would be great if we could pass a custom buffer implementation to the chunked_stream package. I haven't tested this, but I imagine that a Uint8Buffer from package:typed_data will be much more efficient than a general List<int>. We don't even need breaking changes for that, just an optional List<T> Function() createBuffer  parameter that would be used to create internal buffers. I can open a PR for that tomorrow.

If we need/want any modifications to ChunkedStreamIterator let me know

For performance, I think it would be great if we could pass a custom buffer implementation to the chunked_stream package. I haven't tested this, but I imagine that a Uint8Buffer from package:typed_data will be much more efficient than a general List<int>. We don't even need breaking changes for that, just an optional List<T> Function() createBuffer  parameter that would be used to create internal buffers. I can open a PR for that tomorrow.

I would avoid package:typed_data and just use dart:typed_data, recently they even added a BytesBuilder for growing large buffers of bytes.

I would avoid package:typed_data and just use dart:typed_data, recently they even added a BytesBuilder for growing large buffers of bytes.

One thing that makes this problem more problematic is that it's mixing two kinds of stream data.
Most streams are either stand-alone events (often broadcast streams) which can be handled independently, or they're streams of (chunks of) data where you need all of the parts for it to work (often single-subscription streams).
This stream has both: The individual headers, and the chunked content of the following file.
I can see why making it a stream of (header + stream of data) looks useful, but it is problematic because the underlying data comes from the same stream, so you have to skip data to get to the next header. With a "push" approach (streams), there is no good place to say that you don't want the data. It could be a broadcast data stream, but that's fragile and error-prone (you loop over the entries, then asynchronously open a file to pipe data into, and whoops, you missed the first chunk). The alternative is buffering entire files that you don't care about, which is also an issue. (At least they'd be GCed when the file has been read completely, and no-one is hanging onto the stream any more).
That is why I recommend something like StreamIterator, even though it's not a particularly good API either. It's heavily state-based, forcing you to do things in a specific order which isn't visible from the interface.
(You can only read current after moveNext has completed with true and until calling moveNext again. You cannot call moveNext while a moveNext call is waiting to complete. So, every member of the API can throw if you call it at the wrong time, which is why I consider it a problematic API.)
An alternative to StreamIterator<Header> with a content getter (or currentContent) is a StreamIterator<TarFile> with
abstract class TarFile {
  Header get header;
  Stream<Uint8List> get content;
}

That avoids adding an extra content/currentContent member to the interface, it will just be a plain StreamIterator<TarFile>.
Again, that value will only be valid until the next invocation of moveNext. If you call moveNext before starting to read content, it's now too late because we've already started skipping data to get to the next header.
That will have to be well documented (and it's another chance to throw a StateError, yey!)
It's possible to allow you to start reading content, and then call moveNext before the content has been fully read. That just means that we do something with the data we pass by on the way to the next header instead of ignoring it.
Or you can make moveNext throw if you call it while reading content.
Having an internal state machine while processing a structured stream seems unavoidable. Then it's "just" a matter of figuring out which states are needed, and how to transition between them.

One thing that makes this problem more problematic is that it's mixing two kinds of stream data.
Most streams are either stand-alone events (often broadcast streams) which can be handled independently, or they're streams of (chunks of) data where you need all of the parts for it to work (often single-subscription streams).
This stream has both: The individual headers, and the chunked content of the following file.
I can see why making it a stream of (header + stream of data) looks useful, but it is problematic because the underlying data comes from the same stream, so you have to skip data to get to the next header. With a "push" approach (streams), there is no good place to say that you don't want the data. It could be a broadcast data stream, but that's fragile and error-prone (you loop over the entries, then asynchronously open a file to pipe data into, and whoops, you missed the first chunk). The alternative is buffering entire files that you don't care about, which is also an issue. (At least they'd be GCed when the file has been read completely, and no-one is hanging onto the stream any more).
That is why I recommend something like StreamIterator, even though it's not a particularly good API either. It's heavily state-based, forcing you to do things in a specific order which isn't visible from the interface.
(You can only read current after moveNext has completed with true and until calling moveNext again. You cannot call moveNext while a moveNext call is waiting to complete. So, every member of the API can throw if you call it at the wrong time, which is why I consider it a problematic API.)
An alternative to StreamIterator<Header> with a content getter (or currentContent) is a StreamIterator<TarFile> with
abstract class TarFile {
  Header get header;
  Stream<Uint8List> get content;
}

That avoids adding an extra content/currentContent member to the interface, it will just be a plain StreamIterator<TarFile>.
Again, that value will only be valid until the next invocation of moveNext. If you call moveNext before starting to read content, it's now too late because we've already started skipping data to get to the next header.
That will have to be well documented (and it's another chance to throw a StateError, yey!)
It's possible to allow you to start reading content, and then call moveNext before the content has been fully read. That just means that we do something with the data we pass by on the way to the next header instead of ignoring it.
Or you can make moveNext throw if you call it while reading content.
Having an internal state machine while processing a structured stream seems unavoidable. Then it's "just" a matter of figuring out which states are needed, and how to transition between them.

That avoids adding an extra content/currentContentmember to the interface, it will just be a plainStreamIterator`.

This is a nice suggestion, the we don't need to make a public subtype of StreamIterator we could just have a public function:
StreamIterator<TarFile> readTarStream(Stream<List<int>> tarStream) => ...

It's possible to allow you to start reading content, and then call moveNext before the content has been fully read. That just means that we do something with the data we pass by on the way to the next header instead of ignoring it.

In this case it would also be reasonable to make moveNext wait until the full content has been read.
That will cause a deadlock, but unlike the case with await for the infinite blocking won't happen in stream which just ends without providing the rest of the tar entries.
In either case, it's always reasonable to process the entire content before calling moveNext, so throwing a StateError if moveNext is called before the content is read, is also fair -- and perhaps slightly more reliable as it discourages anyone from trying to moveNext without having finished reading the content and handled any errors that occured from reading the content.

That avoids adding an extra content/currentContentmember to the interface, it will just be a plainStreamIterator`.

This is a nice suggestion, the we don't need to make a public subtype of StreamIterator we could just have a public function:
StreamIterator<TarFile> readTarStream(Stream<List<int>> tarStream) => ...

It's possible to allow you to start reading content, and then call moveNext before the content has been fully read. That just means that we do something with the data we pass by on the way to the next header instead of ignoring it.

In this case it would also be reasonable to make moveNext wait until the full content has been read.
That will cause a deadlock, but unlike the case with await for the infinite blocking won't happen in stream which just ends without providing the rest of the tar entries.
In either case, it's always reasonable to process the entire content before calling moveNext, so throwing a StateError if moveNext is called before the content is read, is also fair -- and perhaps slightly more reliable as it discourages anyone from trying to moveNext without having finished reading the content and handled any errors that occured from reading the content.

In this case it would also be reasonable to make moveNext wait until the full content has been read.
That will cause a deadlock, but unlike the case with await for the infinite blocking won't happen in stream which just ends without providing the rest of the tar entries.

It will automatically delay completion until it reaches the next header in the source stream (or the end of the tar archive, then it will just complete with false).
There is no deadlock unless you start reading the content stream and then pause it and never resume. That's no more a deadlock to me than never calling moveNext again. Yes, you can stall a stream if you want to, but you actually have to explicitly ask for it.
And you can even decide to not honor the pause if someone calls moveNext and buffer the rest of the file instead.
So, states:

Initial state: Not listening on source stream yet. No current. No calls to moveNext.

On moveNext: Start listening to source stream, go to Looking for Header.
On cancel: go to Done.

Looking For Header: Pending moveNext. No-one is currently listening for data.

On moveNext, cancel or current: throw StateError
On source data, check for the start of a header, then read that header (might take multiple chunks).
When full header read, pause source stream, complete last moveNext with true, and go to Header Found.
On EOS, complete last moveNext with false. Go to Done.

Header Found: moveNext has completed with true, and has not been called again. No-one is listening on data. Source stream is paused.

On moveNext: Resume source stream and go to Looking for Header. The current current object becomes invalid, and its
content can no longer be listened to, even if it hasn't been listened to before.
On current.content.listen: Resume source stream and go to Reading Data.
The content stream can no longer be listened to (it's single subscription)
On cancel: Cancel source stream, go to Done.

Reading Data: Someone has called current.content.listen and the data is not done yet, and that subscription has not been cancelled.

on moveNext: Go to Reading Data and Looking For Header. (If not respecting pause, resume source stream here if paused).
on data: Forward to listener (up to next header). If data complete, close content stream, pause source stream, and go to Idle.
on EOS: probably a malformed stream, so close content stream with an error, go to Done.
on content pause, pause source stream. Resume on content resume.
on content cancel, pause source stream and go to Idle.

Idle: Source stream paused, noone reading data, no pending moveNext call.

on moveNext: resume source stream, go to Looking For Header.
on cancel: cancel source stream. Go to Done.
on current, give existing value (if any), but stream cannot be listened to again.

Reading Data and Looking For Header. Same as Reading Data except that there is a pending moveNext.

On moveNext, current or cancel: Throw StateError
On data: forward to listener (up to next header). If data complete, close content stream and go to Looking for Header.
On EOS: Probably a malformed source. Close content stream with an error. Complete moveNext with false and go to Done.
On content subscription pause: pause source stream. Resume on content resume. (If not respecting pause, ignore this line)
On content subscription cancel: go to Looking For Header

Done. Source stream read to completion or stream-iterator cancelled. No current.

On moveNext: return a future already completed with false.
On current: throw StateError.
On cancel: do nothing.

That's how I envision it, but obviously I could be missing some details of Tar-decoding. :smile:
Generally, no member of StreamIterator can be called while a moveNext result is pending. When it completes with true, all members can be used until the next call to moveNext (although using moveNext after cancel could also be made an error).
This propagates into the content stream which you can only start reading until the next moveNext, but you can keep reading after a moveNext if you started before. We just need to know whether you care about the data between the two headers or not, as signaled by calling content.listen before moveNext.

In this case it would also be reasonable to make moveNext wait until the full content has been read.
That will cause a deadlock, but unlike the case with await for the infinite blocking won't happen in stream which just ends without providing the rest of the tar entries.

It will automatically delay completion until it reaches the next header in the source stream (or the end of the tar archive, then it will just complete with false).
There is no deadlock unless you start reading the content stream and then pause it and never resume. That's no more a deadlock to me than never calling moveNext again. Yes, you can stall a stream if you want to, but you actually have to explicitly ask for it.
And you can even decide to not honor the pause if someone calls moveNext and buffer the rest of the file instead.
So, states:

Initial state: Not listening on source stream yet. No current. No calls to moveNext.

On moveNext: Start listening to source stream, go to Looking for Header.
On cancel: go to Done.

Looking For Header: Pending moveNext. No-one is currently listening for data.

On moveNext, cancel or current: throw StateError
On source data, check for the start of a header, then read that header (might take multiple chunks).
When full header read, pause source stream, complete last moveNext with true, and go to Header Found.
On EOS, complete last moveNext with false. Go to Done.

Header Found: moveNext has completed with true, and has not been called again. No-one is listening on data. Source stream is paused.

On moveNext: Resume source stream and go to Looking for Header. The current current object becomes invalid, and its
content can no longer be listened to, even if it hasn't been listened to before.
On current.content.listen: Resume source stream and go to Reading Data.
The content stream can no longer be listened to (it's single subscription)
On cancel: Cancel source stream, go to Done.

Reading Data: Someone has called current.content.listen and the data is not done yet, and that subscription has not been cancelled.

on moveNext: Go to Reading Data and Looking For Header. (If not respecting pause, resume source stream here if paused).
on data: Forward to listener (up to next header). If data complete, close content stream, pause source stream, and go to Idle.
on EOS: probably a malformed stream, so close content stream with an error, go to Done.
on content pause, pause source stream. Resume on content resume.
on content cancel, pause source stream and go to Idle.

Idle: Source stream paused, noone reading data, no pending moveNext call.

on moveNext: resume source stream, go to Looking For Header.
on cancel: cancel source stream. Go to Done.
on current, give existing value (if any), but stream cannot be listened to again.

Reading Data and Looking For Header. Same as Reading Data except that there is a pending moveNext.

On moveNext, current or cancel: Throw StateError
On data: forward to listener (up to next header). If data complete, close content stream and go to Looking for Header.
On EOS: Probably a malformed source. Close content stream with an error. Complete moveNext with false and go to Done.
On content subscription pause: pause source stream. Resume on content resume. (If not respecting pause, ignore this line)
On content subscription cancel: go to Looking For Header

Done. Source stream read to completion or stream-iterator cancelled. No current.

On moveNext: return a future already completed with false.
On current: throw StateError.
On cancel: do nothing.

That's how I envision it, but obviously I could be missing some details of Tar-decoding. :smile:
Generally, no member of StreamIterator can be called while a moveNext result is pending. When it completes with true, all members can be used until the next call to moveNext (although using moveNext after cancel could also be made an error).
This propagates into the content stream which you can only start reading until the next moveNext, but you can keep reading after a moveNext if you started before. We just need to know whether you care about the data between the two headers or not, as signaled by calling content.listen before moveNext.

Thanks a lot for the suggestions and the state machine definition. Another benefit of the TarFile class is that the writer can be a Sink<TarFile>.
For now, I think I'll simplify things by disallowing all concurrent reads, so moveNext() would throw if there's an active entry being listened to. That reduces the transitions I need to take care of, the rest is fairly straightforward to implement with ChunkedStreamIterator.substream() + Stream.eventTransformed to detect an unexpected EoS. That should cover most usage scenarios, I can always allow more concurrency later.

Thanks a lot for the suggestions and the state machine definition. Another benefit of the TarFile class is that the writer can be a Sink<TarFile>.
For now, I think I'll simplify things by disallowing all concurrent reads, so moveNext() would throw if there's an active entry being listened to. That reduces the transitions I need to take care of, the rest is fairly straightforward to implement with ChunkedStreamIterator.substream() + Stream.eventTransformed to detect an unexpected EoS. That should cover most usage scenarios, I can always allow more concurrency later.


Error starting file onionservice.yaml

Hello! I use centos7, kubectl version:
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
I am doing a deployment with a fixed address, the points:
Create echoserver.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: http-app
spec:
replicas: 2
selector:
matchLabels:
app: http-app
template:
metadata:
labels:
app: http-app
spec:
containers:
- name: http-app
image: gcr.io/google_containers/echoserver:1.8
ports:
- containerPort: 8080
kubectl apply -f echoserver.yaml
Then
openssl genrsa -out private_key 1024
kubectl create secret generic example-onion-key --from-file=private_key
And
Create an onion service, onionservice.yaml:
apiVersion: tor.k8s.io/v1alpha1
kind: OnionService
metadata:
name: example-onion-service
spec:
version: 2
selector:
app: http-app
ports:
- targetPort: 8080
publicPort: 80
privateKeySecret:
name: example-onion-key
key: private_key
After executing the command: kubectl apply -f onionservice.yaml
I get an error:  error: unable to recognize "./onionservice.yaml": no matches for kind "OnionService" in version "tor.k8s.io/v1alpha1"
Please tell me how can I fix this?

sounds like you haven't installed the custom resource.
What's the output of kubectl get crd?

What's the output of kubectl get crd?
Oh sorry! I did not pay attention to the error when installing the tor-controller and was sure that it was installed. Now I corrected the error, installed the tor-controller, launched onionservice.yaml again. Now everything started perfectly. Thanks!


Make rustc-serialize optional to help WASM targets

rustc-serialize does not build for targets like wasm32-unknown-unknown
and this is probably not going to change, c.f. [1]. To make this crate
usable on these targets without removing functionality, this change makes the
dependency on rustc-serialize optional but default.
[1] https://github.com/rustwasm/team/issues/96

@abaumhauer Anything we can do to help you with this?

Ran into this same issue, thank you for your fix Adam. I'm currently patching dependencies in my project to get this to work with Yew + WASM, would be great if it was mainlined.

Thanks for this PR. Sorry I was away for so long.


Add .npmignore #626

I'm figuring out how to test it... If Windshaft-cartodb tests pass with it then I'll be fine with it.
This closes #626

I think (not 100% sure) that since you declare the .npmignore the .gitignore is no longer taken into account, so any file listed there should be listed in the .npmignore file.

Yep, you are right. Actually the rules are described in the link I posted in the changeset:
https://docs.npmjs.com/misc/developers#keeping-files-out-of-your-package
There are also hints about how to test it.
So, I'll give it some love and then ask for a review when I feel more confident with it. Thanks and sorry.

Testing it:
$ cd Windshaft/
$ npm pack
windshaft-4.8.2.tgz

$ mkdir -p /tmp/test-windshaft-package
$ mv windshaft-4.8.2.tgz /tmp/test-windshaft-package/

$ du -sh windshaft-4.8.2.tgz
92K     windshaft-4.8.2.tgz

It took me a while to figure out how to test it. Turns out that linking the uncompressed package is an option, but you should not forget the yarn install within the package to install its dependencies. Otherwise, gentle nodejs fails silently when it can't find dependencies recursively:
stat("/tmp/test-windshaft-package/package/lib/windshaft/node_modules", 0x7ffda03209b0) = -1 ENOENT (No such file or directory)
stat("/tmp/test-windshaft-package/package/lib/node_modules", 0x7ffda03209b0) = -1 ENOENT (No such file or directory)
stat("/tmp/test-windshaft-package/package/node_modules", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat("/tmp/test-windshaft-package/package/node_modules/grainstore", 0x7ffda03209b0) = -1 ENOENT (No such file or directory)
stat("/tmp/test-windshaft-package/package/node_modules/grainstore.js", 0x7ffda0320940) = -1 ENOENT (No such file or directory)
stat("/tmp/test-windshaft-package/package/node_modules/grainstore.json", 0x7ffda0320940) = -1 ENOENT (No such file or directory)
stat("/tmp/test-windshaft-package/package/node_modules/grainstore.node", 0x7ffda0320940) = -1 ENOENT (No such file or directory)
open("/tmp/test-windshaft-package/package/node_modules/grainstore/package.json", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/tmp/test-windshaft-package/package/node_modules/grainstore/index.js", 0x7ffda0320940) = -1 ENOENT (No such file or directory)
stat("/tmp/test-windshaft-package/package/node_modules/grainstore/index.json", 0x7ffda0320940) = -1 ENOENT (No such file or directory)
stat("/tmp/test-windshaft-package/package/node_modules/grainstore/index.node", 0x7ffda0320940) = -1 ENOENT (No such file or directory)

I prefer the following method:
Apply this patch to Windshaft-cartodb
diff --git a/package.json b/package.json
index d37107b..4495175 100644
--- a/package.json
+++ b/package.json
@@ -49,7 +49,7 @@
     "step-profiler": "0.3.0",
     "turbo-carto": "0.20.2",
     "underscore": "1.6.0",
-    "windshaft": "4.8.1",
+    "windshaft": "file:/tmp/test-windshaft-package/windshaft-4.8.2.tgz",
     "yargs": "11.1.0"
   },
   "devDependencies": {

and then install it:
$ cd Windshaft-cartodb
$ rm -rf node_modules/ && yarn install
$ yarn test

I got some warnings about Dependency="windshaft" installed version="4.8.2" does not match declared version="file:/tmp/test-windshaft-package/windshaft-4.8.2.tgz". Check your installation., but other than that the tests are OK. That gives me confidence in this changeset.


Bump IS-KM connector version

Description
Version of IS-KM connector needs to be bumped to the latest.
This should be reflected in the docs as well related to setting up WSO2 Identity Server as a Key Manager.
A doc issue [2] has been created to track this.
[1] https://apim.docs.wso2.com/en/latest/install-and-setup/setup/distributed-deployment/configuring-wso2-identity-server-as-a-key-manager/
[2] https://github.com/wso2/docs-apim/issues/6658
Affected Component
APIM
Version
4.2.0-Beta
Related Issues
No response
Suggested Labels
No response

Fixed with https://github.com/wso2/product-apim/pull/13157

Doc PR https://github.com/wso2/docs-apim/pull/6750/files


gulp says: Task never defined: default

I used the example from the readme file and, I'm getting this error: Task never defined: default

the example that is written in ES2015

You are probably using the wrong gulp-cli module.  You must be using the latest released version.

I've got these versions:
CLI version 0.4.0
Local version 4.0.0-alpha.2

@AndrewEastwood that is the incorrect version, as I stated above.  Use the correct verison, which can be found at https://www.npmjs.com/package/gulp-cli

I am experiencing the same issue with:
$ gulp -v
[13:37:16] Failed to load external module @babel/register
[13:37:16] Requiring external module babel-register
[13:37:16] CLI version 2.0.1
[13:37:16] Local version 4.0.0

@phun-ky
This is an old issue not related to your problem. Your problem is a configuration error, not a Gulp bug. This tracker is reserved for Gulp bugs. For support, use Stackoverflow or Gitter (see README). Your comment don't have enough context. If this is really a Gulp issue, open a separate issue and use the template issue. These rules are not against you personally but to handle the high volume of issues.

This is my gulpfile.js,
var gulp = require("gulp");
var ts = require("gulp-typescript");
var tsProject = ts.createProject("tsconfig.json");
gulp.task("default", function() {
return tsProject.src().pipe(tsProject()).js.pipe(gulp.dest("dist"))
});
This is the version of gulp,
CLI version: 2.2.0
Local version: 4.0.2
But while running the command "gulp; node dist/app.js" I am getting this error 'Task never defined'
Can you please help me on this


add validate method for positive integers

I had an assignment that required this, saw it wasn't a thing, so wanted to go ahead and make a pull request.
http://codepen.io/anon/pen/jqrpPx?editors=1010
New to the whole Pull Request thing, so if you have any input, let me know! :smiley:

Whoops. Somehow managed to make two separate requests.


add component jsonEditor to schema

has no known special props

@foxriver76
Any reason why this issue is still open?

Added


Employer PAYE reference

PAYE reference number field was recently implemented to validate on 10 characters only, but it can be 11 characters if business was registered after 2001.
See example in national minimum wage social care compliance service: https://www.tax.service.gov.uk/apply-for-social-care-compliance-scheme/capacity-registering

More info:
An employer reference number is a unique combination of letters and numbers, also called an employer PAYE reference, PAYE reference number or just abbreviated to ERN. It is given to a business when it registers with HMRC as an employer, serving to identify the employer for employee income tax and national insurance purposes.
When it first employs someone, the business must register– typically online- as an employer with HMRC, who will issue a welcome pack that includes the employer reference number.
The employer reference number is in two parts. The first part is 3 digits, which identifies the HMRC office number that deals with the company’s PAYE (e.g. 135). The second part of the reference, which follows a forward slash, is the tax office’s reference for the employer itself. (e.g. AB56789).
A business registered as an employer before 2001 will have an employer reference number in the form 135/A56789. Those registered after that date will take the form 135/AB56789.

@jenniferclark I think PAYE reference can actually be 14 characters. I was told it is:

a 3 digit tax office number
a slash
between 1 and 10 characters for the tax office employer reference

This was taken from the RTI specification for employers to send PAYE information to HMRC.

This could be to future proof the reference getting longer?
Jennifer Clark | Senior User Researcher (Live Service and A/B
Testing Programme
Lead)
CDIO - Digital Delivery Centre Newcastle | Room BP9001 | Prudhoe House |
Benton Park View | Newcastle upon Tyne | NE99 1ZZ | 03000 572959 |
<EMAIL_ADDRESS>On 27 November 2017 at 10:43, Steven Proctor<EMAIL_ADDRESS>wrote:

@jenniferclark https://github.com/jenniferclark I think PAYE reference
can actually be 14 characters. I was told it is:

a 3 digit tax office number
a slash
between 1 and 10 characters for the tax office employer reference

This was taken from the RTI specification for employers to send PAYE
information to HMRC.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hmrc/design-patterns/issues/138#issuecomment-347144321,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIlLQ0yg6Y2K2iWY34PVx4JHAzDlDU9oks5s6pI_gaJpZM4QpsvL
.

From service Apply for the National Minimum Wage social care compliance scheme

@jennifer-hodgson This is from Pension Registrations Scheme prototype.

Discussion summary from the hackday

I'm working on documenting this, as part of the references and identifiers pattern, and I'm wondering whether guidance exists as to whether we refer to this particular reference number as PAYE reference or employer reference number (I note that this varies in the examples above). Due to the ambiguity, this feels like something we should mention within the documentation - @stevenaproctor, @jenniferclark - do you have any thoughts on this?

AFIK the employer reference number is something different - it's the AB456 bit after the slash. The entire number, including the digits before the slash, is the PAYE reference number. Agree there needs to be consistency, but I haven't been involved in designing any services that use the term.

@jennifer-hodgson
'Employer PAYE reference' is the official term but it gets shortened to 'PAYE reference'. Only employers have them.
I did some digging and found a list of defined terms from HMRC's online services. These seemed less detailed than I remembered so I went back to the descriptions from 2009. The employer PAYE reference explains the different parts like @jenniferclark mentioned.
Both lists contain information that will help document references and identifiers, including the aliases and alternative names issue.
I think we need to add Accounts Office Reference and Company Registration Number and its other names to our list. They are used a lot in business tax services.

This is very useful - thanks folks! @jenniferclark @stevenaproctor

Some of our users have multiple Employer PAYE reference and do not know which to enter in our services. has anyone else seen this before?

Hi,
My company was registered in 1994 and the ERN only has digits after the / (e.g. numbers only in the format of 999/9999999 with seven digits after the slash).  The number is definitely valid as:

I have confirmed it online via HMRC.
I have confirmed by calling the HMRC service desk.
I've been paying myself and other employees with it since 1994.

However I tend to get it rejected on web forms as they are programmed to expect there to be one or more alphabetic characters immediately after the slash.  Could the guidance be updated to reflect that the number can be entirely numeric?

Hi,
We have an ask in PAYE Income, Allowances, Benefits and Deductions, to ask after the pension provider's PAYE reference. Apparently it's the same format as employers but looking to see if any service has asked for this from the pensions side?


Add new job for APIView task after the build publishes artifact

Test changes.

/azp run net - template - ci

API change check
API changes are not detected in this pull request.

/check-enforcer override


H6053: "devices not belong you","code":400,

Govee Device SKU
H6053
Govee2MQTT Version
2024.01.15-fba50505
Describe the issue
Light not controllable. Device suchsessfully added Govee app and working.
Startup Diagnostics
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
GOVEE_API_KEY=REDACTED
GOVEE_MQTT_HOST=<IP_ADDRESS>
GOVEE_PASSWORD=REDACTED
GOVEE_MQTT_PASSWORD=REDACTED
GOVEE_MQTT_USER=mqttuser
GOVEE_MQTT_PORT=1883
++ cd /app
++ exec /app/govee serve
[2024-01-16T08:52:31 INFO  govee::commands::serve] Starting service. version 2024.01.15-fba50505
[2024-01-16T08:52:31 INFO  govee::commands::serve] Querying platform API for device list
[2024-01-16T08:52:32 INFO  govee::commands::serve] Starting LAN discovery
[2024-01-16T08:52:32 INFO  govee::commands::serve] Waiting 10 seconds for LAN API discovery
[2024-01-16T08:52:42 INFO  govee::commands::serve] Devices returned from Govee's APIs
[2024-01-16T08:52:42 INFO  govee::commands::serve] DreamView (45:AC:D9:39:38:37:2D:13 H6199)
[2024-01-16T08:52:42 INFO  govee::commands::serve]   Platform API: devices.types.light. supports_rgb=true supports_brightness=true
[2024-01-16T08:52:42 INFO  govee::commands::serve]                 color_temp=Some((2000, 9000)) segment_rgb=Some(0..15)
[2024-01-16T08:52:42 INFO  govee::commands::serve]
[2024-01-16T08:52:42 INFO  govee::commands::serve] H6053_4662 (F2:C1:A4:C1:38:0A:46:62 H6053)
[2024-01-16T08:52:42 INFO  govee::commands::serve]   Platform API: devices.types.light. supports_rgb=true supports_brightness=true
[2024-01-16T08:52:42 INFO  govee::commands::serve]                 color_temp=Some((2000, 9000)) segment_rgb=Some(0..15)
[2024-01-16T08:52:42 INFO  govee::commands::serve]
[2024-01-16T08:52:42 INFO  govee::service::http] http server addr is <IP_ADDRESS>:8056
[2024-01-16T08:52:51 WARN  govee::cache] request https://openapi.api.govee.com/router/api/v1/device/scenes status 200: OK: parsing https://openapi.api.govee.com/router/api/v1/device/scenes response: Failed with status 400 Bad Request Bad Request: Request to https://openapi.api.govee.com/router/api/v1/device/scenes failed with code 400 Bad Request devices not belong you. Full response: {"requestId":"uuid","msg":"devices not belong you","code":400,"payload":{"sku":"H6053","device":"F2:C1:A4:C1:38:0A:46:62"}}, will use prior results
[2024-01-16T08:52:52 ERROR govee::hass_mqtt::light] Unable to list scenes for H6053_4662 (F2:C1:A4:C1:38:0A:46:62 H6053): list_scene_names: get_scene_caps: request https://openapi.api.govee.com/router/api/v1/device/scenes status 200: OK: parsing https://openapi.api.govee.com/router/api/v1/device/scenes response: Failed with status 400 Bad Request Bad Request: Request to https://openapi.api.govee.com/router/api/v1/device/scenes failed with code 400 Bad Request devices not belong you. Full response: {"requestId":"uuid","msg":"devices not belong you","code":400,"payload":{"sku":"H6053","device":"F2:C1:A4:C1:38:0A:46:62"}}
[2024-01-16T08:52:52 INFO  govee::service::hass] Wait 2.15s for hass to settle on 43 entity configs
[2024-01-16T08:52:54 INFO  govee::service::hass] MQTT connected with status=CONNACK code 0: Connection Accepted.
[2024-01-16T08:52:54 INFO  govee::service::hass] Home Assistant status changed: offline
[2024-01-16T08:52:54 ERROR govee::hass_mqtt::light] Unable to list scenes for H6053_4662 (F2:C1:A4:C1:38:0A:46:62 H6053): list_scene_names: get_scene_caps: request https://openapi.api.govee.com/router/api/v1/device/scenes status 200: OK: parsing https://openapi.api.govee.com/router/api/v1/device/scenes response: Failed with status 400 Bad Request Bad Request: Request to https://openapi.api.govee.com/router/api/v1/device/scenes failed with code 400 Bad Request devices not belong you. Full response: {"requestId":"uuid","msg":"devices not belong you","code":400,"payload":{"sku":"H6053","device":"F2:C1:A4:C1:38:0A:46:62"}}
Additional Logs
State Attributes:
http: null
iot: null
lan: null
overall: null
platform_metadata:
capabilities:
- alarmType: null
eventState: null
instance: powerSwitch
parameters:
dataType: ENUM
options:
- name: "on"
value: 1
- name: "off"
value: 0
type: devices.capabilities.on_off
- alarmType: null
eventState: null
instance: gradientToggle
parameters:
dataType: ENUM
options:
- name: "on"
value: 1
- name: "off"
value: 0
type: devices.capabilities.toggle
- alarmType: null
eventState: null
instance: brightness
parameters:
dataType: INTEGER
range:
max: 100
min: 1
precision: 1
unit: unit.percent
type: devices.capabilities.range
- alarmType: null
eventState: null
instance: segmentedBrightness
parameters:
dataType: STRUCT
fields:
- dataType: Array
defaultValue: null
elementRange:
max: 14
min: 0
elementType: INTEGER
fieldName: segment
options: []
required: true
size:
max: 15
min: 1
- dataType: INTEGER
defaultValue: null
fieldName: brightness
range:
max: 100
min: 0
precision: 1
required: true
unit: null
type: devices.capabilities.segment_color_setting
- alarmType: null
eventState: null
instance: segmentedColorRgb
parameters:
dataType: STRUCT
fields:
- dataType: Array
defaultValue: null
elementRange:
max: 14
min: 0
elementType: null
fieldName: segment
options: []
required: true
size:
max: 15
min: 1
- dataType: INTEGER
defaultValue: null
fieldName: rgb
range:
max: 16777215
min: 0
precision: 1
required: true
unit: null
type: devices.capabilities.segment_color_setting
- alarmType: null
eventState: null
instance: colorRgb
parameters:
dataType: INTEGER
range:
max: 16777215
min: 0
precision: 1
unit: null
type: devices.capabilities.color_setting
- alarmType: null
eventState: null
instance: colorTemperatureK
parameters:
dataType: INTEGER
range:
max: 9000
min: 2000
precision: 1
unit: null
type: devices.capabilities.color_setting
- alarmType: null
eventState: null
instance: lightScene
parameters:
dataType: ENUM
options: []
type: devices.capabilities.dynamic_scene
- alarmType: null
eventState: null
instance: musicMode
parameters:
dataType: STRUCT
fields:
- dataType: ENUM
defaultValue: null
fieldName: musicMode
options:
- name: Energic
value: 5
- name: Rhythm
value: 3
- name: Spectrum
value: 6
- name: Rolling
value: 4
required: true
- dataType: INTEGER
defaultValue: null
fieldName: sensitivity
range:
max: 100
min: 0
precision: 1
required: true
unit: unit.percent
- dataType: ENUM
defaultValue: null
fieldName: autoColor
options:
- name: "on"
value: 1
- name: "off"
value: 0
required: false
- dataType: INTEGER
defaultValue: null
fieldName: rgb
range:
max: 16777215
min: 0
precision: 1
required: false
unit: null
type: devices.capabilities.music_setting
- alarmType: null
eventState: null
instance: diyScene
parameters:
dataType: ENUM
options: []
type: devices.capabilities.dynamic_scene
- alarmType: null
eventState: null
instance: snapshot
parameters:
dataType: ENUM
options: []
type: devices.capabilities.dynamic_scene
device: F2:C1:A4:C1:38:0A:46:62
deviceName: H6053_4662
sku: H6053
type: devices.types.light
platform_state: null
friendly_name: H6053_4662 Status
Home Assistant Logs
No response
Anything else?
Also working partly with HACS "Govee BT Light Advanced"

Is that device yours, or is it shared with you by someone else?
Can the device be controlled otherwise? The consequence of that warning is that we won't populate the list of effects that are available for the light.

Turns out that this is a BLE-only device and this is Govee's confusing error message about it. See also:

https://github.com/wez/govee2mqtt/issues/92

I'll add a quirk that should prevent this from being added as entities to hass.
You can remove/hide/ignore the entity from hass.


systemd config and hardening

Trying to reduce attack surface on falco with systemd
https://www.freedesktop.org/software/systemd/man/systemd.exec.html
For now, pushed in devel branch of my ansible role.
https://github.com/juju4/ansible-falco/blob/devel/templates/systemd-falco.service.j2
Feel free to comment and test!
(testing into containers is obviously limited...)
Thanks

@juju4 is this something that needs to be added to the falco service if it's installed via RPM?

yes, to add to package and replace init file(for debian or redhat)
https://github.com/falcosecurity/falco/blob/32f8e304eb1bd0ee427b763cdd450fa7d66b27e7/scripts/debian/falco

any details on why wontfix?
change is not a complex one even if more testing is needed. Works fine on my side but obviously not representative.
Thanks

@juju4 the wontfix label had been removed, it was automatically added by our stale bot.

Thanks @fntlnz
How to get forward with this?

So @juju4 if I understood correctly what you are suggesting is that we replace our initd files with systemd files in the two scripts folder we have for debian and rpm.
I agree that this is something that should be in our roadmap, we need to figure out two things:

If the output deb and rpm packages are used to install Falco on old machines (without systemd) we need to maintain both
What are the exact systemd options we want, I also think that we need to setup capabilities as we will do for Kubernetes (https://github.com/falcosecurity/falco/issues/628) - https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Capabilities

@juju4 I'm adding this to the Main project board to prioritize it, hope you don't mind if I change this issue title to be more descriptive. Let me know if I got what you meant!

/kind integrations

/kind feature

@juju4 hey! The current maintainers team can't work on this in the short term, would you like to send a PR with what you have ? It looks like you're very close with your definition and we'd like to have a contribution from you !

Hey @juju4 this is a really cool contribution. Do you mind submitting a PR for it? :)

It does not seem that quick

if using normal debian packaging, that would go through rules file like
https://github.com/Ubuntu-Solutions-Engineering/openstack-deb/blob/master/debian/rules
https://github.com/lxc/lxd-pkg-ubuntu/blob/dpm-xenial/debian/rules
but you have no such file, nor control or changelog in debian dir which are required by dpkg-buildpackage

It seems that you are using cpack debian generator but it has no systemd option that I see
https://cmake.org/cmake/help/v3.6/module/CPackDeb.html
digging more, found this, which might be an option
https://github.com/arangodb/arangodb/blob/master/Installation/debian/postinst.in

quick untested PR


Can't compile project

git clone https://github.com/QuickBlox/q-municate-services-ios.git
Fix framework search paths in Quickblox.xcconfig to ../Frameworks
Replace old Quickblox.framework in Frameworks with latest one.
Add libextobjc library to External/libextobjc
Add libextobjc to QMUsersService.xcodeproj, QMChatService.xcodeproj, add corresponding #imports of libextobjc files.
Fix missing #import <Bolts/Bolts.h> in QMAuthService, QMChatService, and QMContactListService prefix headers.

Now each separate subproject of QMServices project compiles without errors.

Build QMServices target.

Expected result:
It should build successfully after step 1.
Actual result:
Build fails on the line 10 of QMServices.h with error 'Quickblox/Quickblox.h' file not found, however it s there.

Hi,
Does the compiler actually see Quickblox framework?

Of course yes, otherwise after step 6 I wouldn't be able to compile any subprojects as they use Quickblox too.
Have you tried to reproduce step by step?

Have you configured framework search paths for both all QMServices .xcodeproj ?

Yes.

Please show your Podfile content.

Podfile? Here I'm trying to compile your q-municate-services-ios and it doesn't seem to use pods for it's own dependencies.

We will investigate this problem about manual installation. The fastest way for you to make it work for now is to use our development spec on Cocoapods.
For this you need only to configure podfile to look into a specific github branch or even specific commit.
Examples:
pod 'QMServicesDevelopment', :git => 'git@github.com:QuickBlox/q-municate-services-ios.git', :commit => 'ab56ac143bb770e76409a027bfb38de582d728aa'
pod 'QMServicesDevelopment', :git => 'git@github.com:QuickBlox/q-municate-services-ios.git', :branch => 'master'

This way Quickblox framework will not be downloaded as dependency and you will need to manually add it to your own project.

This way after pod install and putting Quickblox.framework to External/ folder in project root, it fails with errors:
![Image of error]
(http://i.imgur.com/khNKbs5.png)
If drag & drop Quickblox.framework into Pods project it builds, but as you know Pods project is regenerated on each pod install.

Have you tried deleting Podfile.lock, Pods folder and .xworkspace file before pod install?

Can't delete Podfile.lock as some pods has no version declaration and are leading to breaking updates.
Delete Pods folder and .xcworkspace file and run pod install end up with the same errors.

I think here is issue with Quickblox.framework as warning states is not a dylib. I've copied it from Frameworks folder of the same commit as my Podfile specifies:
platform :ios, '8.0'
use_frameworks!

# ... Other pods
pod 'QMServicesDevelopment', :git => 'git@github.com:QuickBlox/q-municate-services-ios.git', :branch => 'development', :commit => '533877a6ea527f09ee8bf5b339e6ed00516a78da'

Fixed by hacking your development podspec and adding 'OTHER_LDFLAGS' => "-framework 'Quickblox'" to .xcconfig

Glad to hear you fixed it for yourself, we will investigate this issue and fix manual installation as soon as possible.

Story continues...
After #import <QMServices.h> in bridging header it shows error "'QMServices.h' not found". As I have use_frameworks! in my Podfile, cocoapods is building QMServicesDevelopment as a framework too. So I tried to #import <QMServicesDevelopment/QMServices.h>. Now it fails with "Could not build module 'QMServicesDevelopment'" and bunch of Expected a type errors:
![Image of error]
(http://i.imgur.com/ju6EXwT.png)

@yakubbaev,
We have updated QMServices to 0.3.5 and it is available in master branch now. We have also updated out manual installation guide, please check it out. Try to manual install it using submodule.

 Still.

Fixed by #106

Thank you for contributing to our project! :)
Merged.

getting below issue on QMAssetLoader.m file

@hadwanihardik please create a separate own issue and describe your problem with as much details as possible.


Container dns doesn't work with non-default docker bridge subnet

Currently when we start docker we have a hardcoded --dns parameter set to <IP_ADDRESS> and also the systemd services adds iptables rules to forward DNS traffic going to <IP_ADDRESS> to <IP_ADDRESS>, which is our local dnsmasq resolver.
This is wrong because when the docker daemon starts up it tries to find a free subnet to create its docker0 bridge in. It can happen that the <IP_ADDRESS>/16 subnet is already taken (by wifi for example) and so docker0 will be in one of the other private subnets in IPv4. When this happens the containers will try to resolve using a non-existent DNS server.
 Front conversations

@petrosagg do you have a good idea of how we will fix this? or should we bring it to an arch call?

@shaunmulligan no I don't, and I don't think this is a topic for an arch call. It needs investigation on a device.

@petrosagg could we perhaps use something like the --dns option when starting containers? I guess this would be pretty hacky though

I don't understand the proposed solution. Can you give more details?

after re-reading the docs, I don't actually think its a solution :P  This bug suck, because it happens on both the organge pi and the nanopi always. I don't know why the 172.17.x.x subnet is never taken by the docker bridge. Maybe something new introduced in kernel (both devices run 4.11.5)

@agherzan @petrosagg  I noticed in my kernel 4.11 boot logs of the Nanopi, the following log:
[    0.697806] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.

could this be cause for why docker consistently doesn't attach the bridge on <IP_ADDRESS>

@shaunmulligan I get this on a device which did not get a kernel update iirc.
The only thing that changed was meta-resin

Guess this can be closed now that https://github.com/resin-os/meta-resin/pull/802 is merged?

Yes, closing


Make export::acquire() and export::release() unsafe

Calling export::release() may enable interrupts, which is unsound
if done inside a critical section or other code which expects interrupts
to be disabled.
Calling export::acquire() is less dangerous (no obvious way to cause
unsoundness), but is still a bad idea and breaks the safety contract
of critical_section::acquire(), which must only be called paired with
critical_section::release().

Thank you!
bors r+


Added new 'popupConstraints' property to S2ModalConfig to better support popups on wide devices

Problem:
When using the existing popup dialog on a Flutter web app or Windows app the popup form is displayed with a full screen width. This looks really bad on devices with a large screen width.
Solution:
Have added a new optional 'popupConstraints' property to the S2ModalConfig class.  This allows the modal to be set up with a maximum width and height, that provides a much better visualization on web and windows apps.
Have added a new demo entry in the 'single_popup' class to show how this works.

Hi everyone,
I'm really sorry for not maintaining the smart_select package in a long time. It's been a great project, but it's become too difficult for me to maintain, especially since I've been going through a tough time for the past few years.
In its place, I've released a new package called choice. The combination to smart_select and chips_choice with cleaner, more flexible, and composable API for creating inline or prompted choice widgets with single or multiple selection.
I hope you'll check out choice. I think you'll find it to be a great replacement for smart_select.
Thanks for your understanding.


Agent can try to move out of bounds, throwing an ArrayIndexOutOfBoundsException

https://github.com/jmacglashan/burlap/blob/cc3b09de81f33e6bed8154144a3183dee8595e32/src/main/java/burlap/domain/singleagent/blockdude/BlockDudeModel.java#L138
Because dir can be +1 and ax can be the map.length - 1, the agent can try to move outside the bounds of the map array.
To reproduce, run this test with getLevel1 or getLevel2 instead of getLevel3: https://github.com/jmacglashan/burlap/blob/cc3b09de81f33e6bed8154144a3183dee8595e32/src/test/java/burlap/testing/TestBlockDude.java#L40

any solutions to it instead of modified the source code by my self


lime test windows not working after compiling

Describe your bug here.
i changed the application name and (tried to) change the icons, and after that i tried to compile so it works
but then it hits me with a error PLEASE HELP

Command Prompt/Terminal logs (if existing)
look at linked file

Are you modding a build from source or with Lua?
Source
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
Yes, i edited project.xml so i could change the application name

Delete the export folder and try again

Delete the export folder and try again

are you sure?

Yea
That error happens when something got corrupted when compiling (usually if you close the terminal while it's compiling a file) and just building from scratch fixes it


Grammatical Error

Under FAQ
Q: What happens to my data after the trial expires?
A: After your trial expires, yo have access to your trial data (data from features in Defender for Office 365 that you didn't have previously) for 30 days. After this 30 day period, all policies and data that were associated with the Defender for Office 365 trial are deleted.
Should read "you'll"
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: d08fef08-2fa6-79c4-4adf-e1e194a33a0d
Version Independent ID: 6c962f09-1e5d-bcac-c58f-ecaf58bc9774
Content: Try and evaluate Defender for Office 365
Content Source: microsoft-365/security/office-365-security/try-microsoft-defender-for-office-365.md
Service: microsoft-365-security
Sub-service: mdo
GitHub Login: @chrisda
Microsoft Alias: chrisda

@Phantom-Physics, fixed.
In the future, you too can make changes like this yourself without opening a GitHub issue first. Instructions are here.
Thanks for the feedback.
Closing

Cool!! Good to know. Thank you
On Fri, Jul 14, 2023 at 3:39 PM Chris Davis @.***>
wrote:

Closed #12470
https://github.com/MicrosoftDocs/microsoft-365-docs/issues/12470 as
completed.
—
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/microsoft-365-docs/issues/12470#event-9826189069,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AYOKF7KEHTA5I2FLLDFKI5TXQG4CFANCNFSM6AAAAAA2KZ2KXM
.
You are receiving this because you were mentioned.Message ID:
<MicrosoftDocs/microsoft-365-docs/issue/12470/issue_event/9826189069@
github.com>


Set environment variables for Stagecraft creds

These creds are already defined in the deployment repo so we can make use of them with hiera lookups.

👍


[Bug Report] scaling_factor broke sae_vis compatibility

Describe the bug
I'm using this notebook on an SAE I created: basic_loading_and_analysing.ipynb. I get an error that appears to be because the scaling_factor was added to the SparseAutoencoder class, which sae_vis is not expecting.
Code example
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[21], line 17
     14 print(type(sparse_autoencoder))
     15 print(sparse_autoencoder.state_dict().keys())
---> 17 sae_vis_data_gpt = SaeVisData.create(
     18     encoder=sparse_autoencoder,
     19     model=model,
     20     tokens=all_tokens,  # type: ignore
     21     cfg=feature_vis_config_gpt,
     22 )

File /opt/conda/lib/python3.10/site-packages/sae_vis/data_storing_fns.py:1017, in SaeVisData.create(cls, encoder, model, tokens, cfg, encoder_B)
   1014 # If encoder isn't an AutoEncoder, we wrap it in one
   1015 if not isinstance(encoder, AutoEncoder):
   1016     assert (
-> 1017         set(encoder.state_dict().keys()) == {"W_enc", "W_dec", "b_enc", "b_dec"}
   1018     ), "If encoder isn't an AutoEncoder, it should have weights 'W_enc', 'W_dec', 'b_enc', 'b_dec'"
   1019     d_in, d_hidden = encoder.W_enc.shape
   1020     device = encoder.W_enc.device

AssertionError: If encoder isn't an AutoEncoder, it should have weights 'W_enc', 'W_dec', 'b_enc', 'b_dec'

When running this cell:
from sae_vis.data_config_classes import SaeVisConfig
from sae_vis.data_storing_fns import SaeVisData

test_feature_idx_gpt = list(range(10)) + [14057]

feature_vis_config_gpt = SaeVisConfig(
    hook_point=hook_point,
    features=test_feature_idx_gpt,
    batch_size=2048,
    minibatch_size_tokens=128,
    verbose=True,
)

sae_vis_data_gpt = SaeVisData.create(
    encoder=sparse_autoencoder,
    model=model,
    tokens=all_tokens,  # type: ignore
    cfg=feature_vis_config_gpt,
)

This print statement shows a scaling_factor key that isn't in the above assert:
print(type(sparse_autoencoder))
print(sparse_autoencoder.state_dict().keys())

<class 'sae_lens.training.sparse_autoencoder.SparseAutoencoder'>
odict_keys(['W_enc', 'b_enc', 'W_dec', 'b_dec', 'scaling_factor'])

System Info
Describe the characteristic of your environment:

Describe how transformer_lens was installed (pip, docker, source, ...) pip
What OS are you using? (Linux, MacOS, Windows) Ubuntu
Python version (We suppourt 3.10 -3.12 currently) 3.10

Checklist

[x] I have checked that there is no similar issue in the repo (required)

Hey Adam,
Thanks for raising this. We're moving pretty quickly so it's easy for things to get out of sync. We did catch this and will be implementing integration tests between SAE Vis and SAE Lens shortly.
If you update sae_vis this should be fixed. (https://github.com/callummcdougall/sae_vis/blob/d759ef0237089e72cc9cad7edc4eceb4e8cfdd00/sae_vis/data_storing_fns.py#L1026)
A bit more detail: SAE Vis currently makes strong assumptions about the forward pass of the Autoencoder which won't in general be true (eg: pre-encoder subtraction of the decoder bias, relu activation etc). So I think we'll need to find a solution for this (likely creating some spec which allows SAE Vis to black box the Autoencoder).


Callback for 'ask' command missing 'ctx' parameter.

Summary
The confirm button example is incorrect - it does not work.
Reproduction Steps
Copy and pasted disnake/examples/views/button/confirm.py in two different files - the class and the command in two different files.
Then, I imported the file (and class). When I ran the command, I got an error.
Minimal Reproducible Code
import disnake
from disnake.ext import commands

class Confirm(disnake.ui.View):
    def __init__(self):
        super().__init__()
        self.value = None

    # When the confirm button is pressed, set the inner value to `True` and
    # stop the View from listening to more input.
    # We also send the user an ephemeral message that we're confirming their choice.
    @disnake.ui.button(label="Confirm", style=disnake.ButtonStyle.blurple)
    async def confirm(ctx: commands.Context, self, button: disnake.ui.Button, interaction: disnake.MessageInteraction):
        await interaction.response.send_message("Confirming", ephemeral=True)
        self.value = True
        self.stop()

    # This one is similar to the confirmation button except sets the inner value to `False`
    @disnake.ui.button(label="Cancel", style=disnake.ButtonStyle.grey)
    async def cancel(ctx: commands.Context, self, button: disnake.ui.Button, interaction: disnake.MessageInteraction):
        await interaction.response.send_message("Cancelling", ephemeral=True)
        self.value = False
        self.stop()

# This is in another file

    @bot.command()
    async def ask(ctx: commands.Context):
      """Asks the user a question to confirm something."""
      # We create the view and assign it to a variable so we can wait for it later.
      view = Confirm()
      await ctx.send("Do you want to continue?", view=view)
      # Wait for the View to stop listening for input...
      await view.wait()
      if view.value is None:
          print("Timed out...")
      elif view.value:
          print("Confirmed...")
      else:
          print("Cancelled...")

Expected Results
I expected the buttons to be made. However, it gave me an error.
Actual Results
I got a traceback error.
Traceback Error
Ignoring exception in on_message
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/disnake/ext/commands/core.py", line 752, in _parse_arguments
    next(iterator)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/disnake/client.py", line 505, in _run_event
    await coro(*args, **kwargs)
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/disnake/ext/commands/bot_base.py", line 600, in on_message
    await self.process_commands(message)
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/disnake/ext/commands/bot_base.py", line 597, in process_commands
    await self.invoke(ctx)
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/disnake/ext/commands/bot_base.py", line 560, in invoke
    await ctx.command.invoke(ctx)
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/disnake/ext/commands/core.py", line 921, in invoke
    await self.prepare(ctx)
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/disnake/ext/commands/core.py", line 855, in prepare
    await self._parse_arguments(ctx)
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/disnake/ext/commands/core.py", line 754, in _parse_arguments
    raise disnake.ClientException(
disnake.errors.ClientException: Callback for ask command is missing "ctx" parameter.

Intents
all intents
System Information
- Python v3.8.12-final
- disnake v2.2.2-beta
    - disnake pkg_resources: v2.2.2
- aiohttp v3.7.4.post0
- system info: Linux 5.11.0-1023-gcp #25~20.04.1-Ubuntu SMP Mon Nov 15 15:54:39 UTC 2021

Checklist

[X] I have searched the open issues for duplicates.
[X] I have shown the entire traceback, if possible.
[X] I have removed my token from display, if visible.

Additional Context
No response

You forgot to add a self parameter to the view methods.

You forgot to add a self parameter to the view methods.

Thank you - Maybe the examples should be updated to include this - I copy-pasted directly from the examples.

You forgot to add a self parameter to the view methods.

Thank you - Maybe the examples should be updated to include this - I copy-pasted directly from the examples.

I think you should teach OOP, that's all 😳👍


feat(ci,docker): save RUN --mount=type=cache ccache on GitHub Actions cache

Description

[ ] https://github.com/autowarefoundation/autoware/pull/4842

I've finally understood the BuildKit doesn't preserve cache mounts in the GitHub Actions cache by default. If we wish to put our cache mounts into GitHub Actions cache and reuse it between builds, we can use a workaround provided by reproducible-containers/buildkit-cache-dance
https://docs.docker.com/build/ci/github-actions/cache/#cache-mounts
I found that the buildkit-cache-dance action does not work effectively unless the setup-buildx-action action is executed beforehand, so it was temporarily moved from docker-build-and-push/action.yaml to the beginning of each workflow file. This will be improved in the next PR.
Tests performed
~Currently, as part of autoware.universe is broken,~ I show the build results of only the core module.
https://github.com/youtalk/autoware/actions/runs/9462979885?pr=44

1st try (no cache):

docker-build-and-push-main (no-cuda) 10m 7s
docker-build-and-push-main (cuda) 16m 9s

2nd try (with cache):

docker-build-and-push-main (no-cuda) 4m 17s
docker-build-and-push-main (cuda) 5m 27s

Effects on system behavior

Not applicable.
Interface changes

Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[x] The PR follows the pull request guidelines.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

@youtalk I think after this PR was merged, docker-build-and-push-main-self-hosted.yaml started failing.

https://github.com/autowarefoundation/autoware/actions/runs/9477822140/job/26113063357#step:9:859
  /usr/bin/docker buildx bake --file docker/docker-bake.hcl --file /home/ubuntu/actions-runner/_work/_temp/docker-actions-toolkit-MB8IMw/docker-metadata-action-bake.json --file /home/ubuntu/actions-runner/_work/_temp/docker-actions-toolkit-Jw8xni/docker-metadata-action-bake.json --file /home/ubuntu/actions-runner/_work/_temp/docker-actions-toolkit-zQNIl8/docker-metadata-action-bake.json --set *.platform=linux/arm64 --set *.args.ROS_DISTRO=humble --set *.args.BASE_IMAGE=ros:humble-ros-base-jammy --set *.args.SETUP_ARGS= --set *.args.LIB_DIR=aarch64 --set *.cache-from=type=registry,ref=ghcr.io/autowarefoundation/autoware-buildcache:cuda-arm64-main --set *.cache-from=type=registry,ref=ghcr.io/autowarefoundation/autoware-buildcache:cuda-arm64-main --set *.cache-to=type=registry,ref=ghcr.io/autowarefoundation/autoware-buildcache:cuda-arm64-main,mode=max --provenance false --push --print
  unknown flag: --provenance
  See 'docker buildx bake --help'.

Do I need to install something to the arm64 machine to make this work? It is a self hosted machine and I have access to it.

I've updated the machine and maybe it will start working again :man_shrugging:
https://github.com/autowarefoundation/autoware/actions/runs/9516463540

@xmfcx Thank you! I think we need to update docker and you did it.

Yes, it seems the system upgrade solved it. I've also restarted and reinstalled the actions runner to have it clean.


UserID as data column

For ease of retrieval, it would be great if UserID is used for getting tweets (instead of only username).
Same can be said for TweetID, reply/quote TweetID etc.

User IDs have been added!

Nice!


Querydsl-core: medium severity vulnerability found on<EMAIL_ADDRESS>

According to Snyk securityscan of Querydsl 4.3.1, there is a medium severity vulnerability found on<EMAIL_ADDRESS>[WARNING] ✗ medium severity vulnerability found on<EMAIL_ADDRESS>[WARNING] - desc: Deserialization of Untrusted Data
[WARNING] - info: https://snyk.io/vuln/SNYK-JAVA-COMGOOGLEGUAVA-32236
[WARNING] - from<EMAIL_ADDRESS>><EMAIL_ADDRESS>See remediation:
https://snyk.io/vuln/SNYK-JAVA-COMGOOGLEGUAVA-32236

Do don't have to use Guava 18.0, as long as any version of Guava is provided it is fine. We require 18.0 minimally. So its recommended to just bump the version  of Guava in your project.


Datepicker popup's initial format is broken in angular 1.5.0

Please refer to https://github.com/angular-ui/bootstrap/issues/3025
Commented by rustygreen: "The datepicker popup's initial display value does not use the passed in format. After you select a value for the first time, it will display it property (using the passed in format)."
Angular 1.4.8 works fine but 1.5.0:
http://plnkr.co/edit/cg1Uj5kF8fNAaNuTvxVY?p=preview

This appears to be an issue with current master as well, as can be seen here.

Looks like a workaround for now is to use init-date as the same value as your model, as shown here - need to investigate further though.


Provide a more gulpy gulp interface

gulp.task('test', function() {
  return gulp.src('test/unit/').pipe(wct({...}));
});

May be able to have the test() function behave nicely with streams

I may spend some time and try to make a true gulp plugin for this soon, one question though.
Do you want to keep the plugin within the web-component-tester repo or would it make sense to break it out into a separate gulp-wct plugin which requires web-component-tester?

Separate plugin would probably work best!
On Thu, Aug 20, 2015, 18:38 Gregory Benner<EMAIL_ADDRESS>wrote:

I may spend some time and try to make a true gulp plugin for this soon,
one question though.
Do you want to keep the plugin within the web-component-tester repo or
would it make sense to break it out into a separate gulp-wct plugin which
requires web-component-tester?
—
Reply to this email directly or view it on GitHub
https://github.com/Polymer/web-component-tester/issues/126#issuecomment-133240420
.


Add a warning message when notify payloads are subject to truncation

Goal

Inform the devs when truncation can occur from large payloads

Design

Changeset

Testing

Hi @ChanChar
Thanks for raising this PR. We will make sure to take a look at this when priorities allow, and will post any updates on this thread.


fix: polish new time preview w/ shifting arrow

with the new MC v3 the timerange preview required some small UI updates for the shifting arrow.
also adds some polish discussed in our doc Mux player polishing.

Looks 👌. I'll update my other branch to omit the version upgrade changes if/when you merge this.


feat: support VSCode insiders

This PR allows users to open files in VSCode Insiders instead of regular VSCode.
When "vscode.open.insidersMode" is set to true in User Setting, this extension will open the file in VSCode Insideres using vscode-insiders://file/Users/USERNAME/directory/sourcegraph-open-in-vscode/prettier.config.js vs. vscode://file/Users/USERNAME/directory/sourcegraph-open-in-vscode/prettier.config.js if insidersMode is not set or set to false.
RE: feature request https://github.com/sourcegraph/sourcegraph-open-in-vscode/issues/18

I know yarn.lock file was broken but we still want to keep it around. We want to make sure it's updated though. See why this file is useful.

@muratsu oh gotcha, that was my bad. I will add the new yarn.lock file back


Show different disability filters by district

Who is this PR for?
NB K8 educators
What problem does this PR fix?
New Bedford uses different values than Somerville to describe levels of disability in special education.  This makes the summary values in the overview page inaccurate, and the filter values are not all meaningful.
What does this PR do?
Threads the server env value down to window.shared.Env.districtKey and then branches on it in the SlicePanels UI to show different filters for each district.
Screenshot (if adding a client-side feature)
before, showing wrong filters:

after, showing correct filters (but for demo data using Somerville disability values):

Checklists

[x] Author checked latest in IE - School Overview
[x] Author included specs for new code

selfie


Feature instagram widget

Responsive Instagram component with placeholder images.
Takes an array of URLs representing the hrefs of the Instagram images and displays the first 4 images of the array (3 in mobile view).

I have implemented the changes suggested, let me know if I missed something or if there is anything else you think we should modify.

Looks good with the updates. I took the liberty of adding some hover effects.


3.1.0 re3 room 11b npc graphic glitch

Version
3.1.0
Seed
R730-M01E-ZTM885XZ3XY20QNHMHL20C00H200H200ZZEXZYZZZZ
Scenario
RE 3 (Jill)
Player Character
kendo
What happened?
graphic glitch :
11B:0 (0x4D42) [NIKOLAI ZINOVIEV] becomes [CHIEF IRONS (david)]
11B:0 (0x4E00) [NIKOLAI ZINOVIEV] becomes [CHIEF IRONS (david)]
bug 03_2023-08-08 01^13.18.zip
game ok if disable mod_biorand\room

@otamega72 you are playing an old version. This should be fixed in 3.1.1.


Move the Localnet docker section into separate chapter

Similarly to how we moved the network settings to a separate chapter and how we revamped the Band protocol (#907), I suggest we devote a new "Localnet Development" (dapp/tools/localnet.md) chapter which includes the current content of https://docs.oasis.io/dapp/sapphire/guide#running-a-private-oasis-network-locally
TODO:

[x] mention it could also be used for running Emerald (and replace https://docs.oasis.io/dapp/emerald/writing-dapps-on-emerald#running-a-private-oasis-network-locally with a link to the new chapter)
[ ] mention it in the ROFL chapter: https://docs.oasis.io/rofl/app#testing-it-on-sapphire-localnet
[ ] replace any links throughout the docs pointing to Sapphire Guide subsection or any links pointing directly to https://github.com/oasisprotocol/oasis-web3-gateway/pkgs/container/sapphire-localnet or https://github.com/oasisprotocol/oasis-web3-gateway/tree/main/docker with the new https://docs.oasis.io/dapp/tools/localnet
[x] mention that minimum requirements for running the Sapphire Localnet image (at least 16GB of memory)

Merge sequence:

[ ] New page goes in first https://github.com/oasisprotocol/docs/pull/977
[ ] Dependent reference https://github.com/oasisprotocol/sapphire-paratime/pull/429
[ ] Dependent reference https://github.com/oasisprotocol/oasis-sdk/pull/2023
[ ] Bump sapphire-paratime and oasis-sdk submodules to publish


Type declarations for setProvidersList and Provider

Hello,
In order to use the setProviderList function type definitions for Provider and setProvidersList had to be added.
Issue #66 is related.
:)

@MarcMcIntosh thank you. This will be merged now.


get("labels") is broken

import shogun as sg
from shogun import SVMLight
from shogun import BinaryLabels

num = 10
dist = 1
width = 2.1
C = 2.0
traindata_real = concatenate((randn(2,num)-dist, randn(2,num)+dist), axis=1)
testdata_real = concatenate((randn(2,num)-dist, randn(2,num)+dist), axis=1);

trainlab = concatenate((-ones(num), ones(num)));
testlab = concatenate((-ones(num), ones(num)));

feats_train = sg.features(traindata_real);
feats_test = sg.features(testdata_real);
kernel = sg.kernel("GaussianKernel", log_width=width)
kernel.init(feats_train, feats_train)

labels = BinaryLabels(trainlab);

svm = SVMLight(C, kernel, labels)
svm.train()
print(svm.get_labels())
print(svm.get("labels"))

the last command will fail... this is consistent with c++ api

same is true for LibSVM

The issue is probably because the function in https://github.com/shogun-toolbox/shogun/commit/9537f595679993bd56a833629cba832c97165c5c#diff-1689b0bd3ceb28355cc3c05ed7ffe4f0R1100 doesnt have CLabels (or CDistance) in it

Ah marvelous... thnx for the quick reply... although it doesnt explain why a deser svm fails with not having labels :s (get_labels do returns the labels)

On 14 Apr 2019, at 11:34, Gil<EMAIL_ADDRESS>wrote:
The issue is probably because the function in 9537f59#diff-1689b0bd3ceb28355cc3c05ed7ffe4f0R1100 doesnt have CLabels (or CDistance) in it
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

what do you mean? What error do you get?

That labels are not set (when trying to run train), but as said gat_labels returns a correct binarylabels obj

On 14 Apr 2019, at 12:51, Gil<EMAIL_ADDRESS>wrote:
what do you mean? What error do you get?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

i'll leave this one open until the getter is not added to the interface

is this fixed?

idk, but I guess it would be easy to find out ...