101 Reactive Gems (working title)

In this issue, we should collect tips and tricks with reactive systems and dataflows.
These are not particularly advanced topics but the markdown support on GitHub makes it easier to write them up.
Once we run out of ideas, we may tidy it up and release it together (maybe a free ebook?).

2) compose at subscription time
Modern reactive libraries offer you fluent conversion operators: extend, to, as or compose. You can then apply your own transformative function which runs in assembly time and let's you customize a sequence with preset operators:
Function<Publisher<T>, Publisher<T>> addSchedulers = o -> o.subscribeOn(Schedulers.io()).observeOn(AndroidSchedulers.mainThread());

range(1, 10).compose(addSchedulers).subscribe(System.out::println);

The function addSchedulers is executed when the range.compose is executed in assembly time.

However, sometimes you want stateful operators with a sequence, for example, a counting `map` that updates a "global" state. Clearly, such operation doesn't properly work with multiple concurrent `Subscriber`s to a sequence.

Luckily, you can use the composition operators above in conjunction with `defer` to shift the execution of the function to subscription time and allowing per-`Subscriber` state in the applied operators:

```java
range(1, 10).compose(o -> defer(() -> {
    int[] counter = new int[] { 0 };
    return o.map(v -> ++counter[0]);
})).subscribe(System.out::println);

3) FlatMap as map or filter
The operator flatMap very versatile and comes up frequently when one wants to join different mapped sequences.
The mapper function let's you return any Publisher instance, including empty or single element (just).
We can utilize this property and implement map with flatMap:
Function<T, R> mapper = ...

map(mapper) == flatMap(v -> just(mapper.apply(v)));

Given the current value from source, we apply the mapper function and return a constant just Publisher.
We can apply the same methodology with filter: return empty if the predicate doesn't hold and return just if it does hold:
Predicate<T> predicate = ...

filter(predicate) == flatMap(v -> predicate.test(v) ? just(v) : empty());

Note, however, that using flatMap as such incurs more overhead than a direct map or filter. Modern reactive libraries still optimize for these cases because flatMap let's you short-circuit a sequence by emitting an error Publisher:
source.flatMap(v -> {
    try {
        performIO(v);
        return just(v);
    } catch (IOException ex) {
        return error(ex);
    }
})

4) Continuations with concat
Sometimes, you want to run through a sequence of values and then resume with another sequence of a completely different type once the first completed. In this case, you'd usually also ignore the first sequence's values.
Some libraries already offer operators for it: then, andThen, etc. but if not available, you can use concat for it:
Publisher<Integer> a = ...
Publisher<String> b = ...

Publisher<String> c = (Publisher<String>)concat(a.ignoreElements(), b);

Unfortunately, you'll need some explicit casts and @SuppressWarnings for this; Java's type system doesn't seem to be powerful enough to express it otherwise.

5) subscribeOn and observeOn/publishOn act sometimes the same
One of the confusions with reactive operators are centered around subscribeOn and observeoOn (publishOn). For short, subscribeOn acts during subscription time over subscribe() calls and its effects travel upstream whereas observeOn acts during runtime over onNext|onError|onComplete calls and its effects travel downstream.
However, many sequential data sources, such as just, empty, error, fromArray, fromIterable and range, applying either subscribeOn or observeOn will yield the same output: events signalled on the specified Scheduler's thread.
just(1).subscribeOn(scheduler) == just(1).observeOn(scheduler)

range(1, 5).subscribeOn(scheduler) == range(1, 5).observeOn(scheduler)

fromArray("a", "b", "c").subscribeOn(scheduler) == fromArray("a", "b", "c").observeOn(scheduler)

// etc.

The reason for this equivalence is that these constant sources don't really have subscription side-effects and requests to them will trigger emissions on the Scheduler's thread anyway.
Most modern libraries already exploit this in respect to just; both patterns are transformed into a custom scheduled Publisher instance.
In addition, using observeOn may give better performance with the multi-valued sources due to micro-operator-fusion in its front. However, we don't fully see the ramifications of an automatic subscribeOn -> observeOn swap when a sequence is assembled due to the so-called strong pipelining effect when both operators are applied to the sequence:
range(1, 10).subscribeOn(schedulerA).observeOn(schedulerB);

6) FlatMap as concatMap
The operator flatMap let's you run multiple sources at once while concatMap runs one sequence at a time.
However, flatMap usually offers a parameter that let's limit its concurrency level. The smallest allowed concurrency level is 1, which will functionally act the same as concatMap:
flatMap(mapper, 1) == concatMap(mapper)

This identity helped many times to rule out possible bugs in either flatMap's or concatMap's implementation as a cause of some weird behavior or exception.
Note that in many implementations of concatMap, the operator prefetches its source values (2 elements in RxJava, 32 in Reactor) whereas a concurrency-constrained flatMap will prefetch exactly one source value and thus trigger sequence dependent side-effects at different times.

7) Parallel processing
Reactive datasources are sequential in nature; they form a processing pipeline that runs in FIFO order, even if source values are available while processing earlier values.
Still, sometimes you want to go parallel, for example, execute many service calls at once or distribute some processing over the available CPUs.
There are some experiments with such reactive stream parallelization, but the effect itself can be achieved today with common operators.
Using flatMap
One of the ways is to use flatMap for the job:
range(1, 1_000).flatMap(v -> just(v).subscribeOn(computation).map(Object::hashCode))

If there are a lot of source values, scheduling them one-by-one adds quite an overhead. Instead, you can buffer some of them and act on them in a batched fashion:
range(1, 1_000_000).buffer(256)
.flatMap(list ->
    just(list).subscribeOn(computation).map(v ->
        v.stream().map(Object::hashCode).collect(toList()))
)
.flatMapIterable(c -> c)

If the final order still matters, you can use concatMapEager instead of flatMap.
Using window
The operator window creates a nested Publisher<Publisher<T>> sequences which can be batched similarly to buffer but the values themselves become immediately available in the inner Publisher.
We can use this operator with some mapping and flattening to get the parallel effect:
range(1, 1_000_000).window(256).map(w ->
    w.observeOn(computation).map(Object::hashCode))
.concatMapEager(w -> w);

Note the use of observeOn here: the source inner window is no longer constant like above but emits onNext events as they arrive from the source. Applying subscribeOn would be useless.
Using groupBy
The operator groupBy also creates nested Publisher instances based on some key selector to determine which value goes into which group; i.e., routing or dispatching values to various inner windows.
One way is to use the current value's hashCode module some integer to select a "bucket" for it:
range(1, 1_000_000)
.groupBy(v -> v.hashCode() % 8)
.flatMap(g -> g.observeOn(computation).map(Object::hashCode))

Given a well distributed hashCode, this should yield a fairly balanced parallel processing of values.
Alternatively, you can do round-robin dispatching by using a counter instead:
long[] counter = { 0 };

range(1, 1_000_000)
.groupBy(v -> (counter[0]++) & 7)
.concatMapEager(g -> g.observeOn(computation).map(Object::hashCode))

7) zipWith as a closing companion
The operator instance.zipWith(companion) will run a zipping over the instance source and the companion source.
Once one source is complete, the resulting Publisher completes. That can be taken as an advantage to drive instance with some limit. For instance :
range(1, 1_000_000).zipWith(timer(100, TimeUnit.MILLISECONDS), (a, b) -> a)
//emit for 100ms then cancel the main range if not finished

This identity helped many times to rule out possible bugs in either flatMap's or concatMap's implementation as a cause of some weird behavior or exception.
Note that in many implementations of concatMap, the operator prefetches its source values (2 elements in RxJava, 32 in Reactor) whereas a concurrency-constrained flatMap will prefetch exactly one source value and thus trigger sequence dependent side-effects at different times.

8) Executing an action if the source is empty
Sometimes you want to execute some action, such as logging, when it turns out the source Publisher is empty.
You should be familiar with the switchIfEmpty operator which switches to a new Publisher if the main Publisher is empty. We can then switch to an empty publisher and use doOnComplete with the desired action:
Publisher<T> source = ...

source.switchIfEmpty(empty().doOnComplete(() -> System.out.println("Empty source!")));

9) Processing elements pairwise
Sometimes you want to process subsequent elements of a Publisher pairwise (in triplets, etc.). A possible way of doing this is by using buffer with skip option:
range(1, 9)
.buffer(2, 1)
.filter(b -> b.size() < 2)
.map(b -> b.get(0) + b.get(1))
.subscribe(System.out::println);

Since buffer may emit a partial buffer if the number of source items is odd, you have to filter out partial buffers (otherwise, the map will blow up with IndexOutOfBoundsException)
An alternative way is to use the publish overload taking a Function<Publisher<T>,Publisher<R>>. This overload, unlike the regular publish returns a Publisher and is in fact a cold operator. What it does is that for the duration of a Subscriber, it makes the source into a hot Publisher and you can attach as many operators to it via the Function callback; thus, the source events are shared among different "paths" without subscribing to the source multiple times.
We can use this publish with skip and zip to get pairs:
range(1, 9)
.publish(o -> zip(o, o.skip(1), (a, b) -> a + b))
.subscribe(System.out::println);

There is no need for filtering as there is no list anymore and we get the pairs nicely via lambda parameters. Naturally, it works with triplets as well:
range(1, 9)
.publish(o -> zip(o, o.skip(1), o.skip(2), (a, b, c) -> a + b + c))
.subscribe(System.out::println);

10) Executing a function asynchronously
You may often want to execute some Supplier or Runnable/Action0 asynchronously. Maybe it's something blocking or just long running and don't want to block/hold up your current thread.
Many modern libraries offer the fromCallable(Callable<T>) source factory where you can use a regular old Callable to execute some action. With Java 8 lambdas, it's easy to convert your callbacks into Callables:
Supplier<Integer> supplier = () -> 1;
fromCallable(supplier::get).subscribeOn(computation);

Runnable run = () -> System.out.println("Hello!");
fromCallable(() -> { run.run(); return null; }).subscribeOn(computation);

If for some reason fromCallable is not available, you can use a combination of just and map to get the same effect:
just("whatever")
.subscribeOn(computation)
.map(ignored -> supplier.get());

Of course, you can swap the order of subscribeOn and map here; they are functionally equivalent. However, as we saw with Gem # 5, just().subscribeOn() is usually optimized in modern libraries and an intermediate just().map().subscribeOn() would most likely prevent some optimizations.

11) Caching the last element and clearing it on demand
A possible use for BehaviorSubject/BehaviorProcessor, which remembers the last value and emits it to new Subscribers at the beginning of a sequence, is to use it as a single element cache - also known as a "reactive property".
Sometimes, the contents of this cache can become outdated and shouldn't be emitted to new Subscribers until a proper fresh value is generated. Unfortunately, most implementations don't offer a clear() method so until a new value gets assigned to to it via onNext the old one stays.
We will use this onNext in fact to clear out the current value. To do this, we have to establish a protocol where a special value from the Subject's value type indicates emptiness and instructs Subscribers to ignore it if encountered:
Integer CLEAR = new Integer(0);

BehaviorProcessor<Integer> cache = new BehaviorProcessor<>(CLEAR);

Publisher<Integer> front = cache.filter(v -> v != CLEAR);

front.subscribe(System.out::println);

cache.onNext(10);
cache.onNext(CLEAR);
cache.onNext(20);

front.subscribe(System.out::println);

In the example, we create a new (!) Integer instance and use it for reference comparison to determine if the cache is "empty".
Sometimes, however, you can't just create such an "empty" instance of the type you are working with. In this case, you have to revert to the lowest common denominator type: Object and downcast anything else back to your type that isn't the indicator:
Object CLEAR = new Object();

BehaviorProcessor<Object> cache = new BehaviorProcessor<>();

Publisher<String> front = cache.filter(v -> v != CLEAR).cast(String.class);

cache.onNext("abc");
cache.onNext(CLEAR);
cache.onNext("def");

front.subscribe(System.out::println);

Of course, directly exposing BehaviorSubject<Object> is a welcoming sign for all kinds of types. You can re-establish the type safety by wrapping the code into some class and allowing only type-correct onNext calls:
public final class Cache<T> implements Observer<T>, Publisher<T> {
    static final Object CLEAR = new Object();
    final BehaviorProcessor<Object> cache = new BehaviorProcessor<>(CLEAR);

    @Override
    public void subscribe(Subscriber<? super T> subscriber) {
        cache.filter(v -> v != CLEAR).subscribe(subscriber);
    }

    @Override
    public void onNext(T t) {
        cache.onNext(t);
    }

    @Override
    public void onError(Throwable t) {
        cache.onError(t);
    }

    @Override
    public void onComplete() {
        cache.onComplete();
    }

    public void clear() {
        cache.onNext(CLEAR);
    }
}

Remember, Processors (and Subject) require a non-concurrent calls to their onXXX methods so you should only call them in a serialized fashion (such as a GUI thread or single threaded event-loop).

12) processing elements of list-based values
Sometimes, an API gives you a Publisher<List<T>> where the signaled values are Lists and you want to process the elements in the list while holding the list together to be processed as a whole downstream later on.
Let's start out as a common source of Lists:
Publisher<List<Integer>> source = range(1, 1_000_000).buffer(256);

In-place processing
You can just simply run a for loop and manipulate the list in-place if the list is mutable (buffer emits such mutable list):
source.map(list -> {
    for (int i = 0; i < list.size(); i++) {
        list.set(i, list.get(i) + 2_000_000);
    }
    return list;
});

Or as a new list:
source.map(list -> {
    List<Integer> newList = new ArrayList<>(list.size());
    for (Integer v : list) {
        newList.add(v + 2_000_000);
    }
    return newList;
});

Stream processing
You can combine the worlds of reactive and the interactive Java 8 Stream processing:
source.map(list ->
    list.stream()
    .map(v -> v + 2_000_000)
    .collect(toList())
);

ConcatMap
If you can't mute the list, stuck on Java 7 or before, or just don't want to look "non functional", you can use concatMap (not flatMap in order to keep the original list order), extract each list, process elements and then recollect the result into another list:
source.concatMap(list ->
    fromIterable(list)
    .map(v -> v + 2_000_000)
    .toList()
);

13) Awaiting completion of many sources
Sometimes you have a bunch of Publishers doing various things and you want to wait until they all complete and their emitted values are irrelevant in your case.
You may think of zip or combineLatest for doing this along with ignoreElements, however, zip and combineLatest won't work with most source combinations as you'd expect. These operators are terminate eagerly: if one of them completes without elements, they complete immediately and cancel the rest of the outstadning source sequences. In addition, if zip encounters a shorter sequence, it will also terminate eagerly.
You can simply use flatMap in conjunction with ignoreElements() to join the sources' termination:
Publisher<String> source1 = just("Hello");
Publisher<Integer> source2 = range(1, 3);
Publisher<Long> source3 = timer(1, TimeUnit.MILLISECONDS);

just(source1, source2, source3)
.flatMap(o -> o.ignoreElements())
.doOnComplete(() -> System.out.println("Done"))

14) Errors as non-terminal events
By default, reactive protocols treat onError events as fatal and terminal events, tearing down the whole chain you delicately assembled.
Many times, you'd want to treat an error like any other value and keep the sequences running. For this, you have to hide an error from the library by first not calling onError but create a holder class for both normal and error values and emit those. Libraries often support this via classes such as Notification<T> or Signal<T>, very similar to Java 8's Optional<T> but for values, errors and emptiness:
PublishSubject<Notification<Integer>> bus = PublishSubject.create();

bus.dematerialize().subscribe(System.out::println, Throwable::printStackTrace);
bus.subscribe(System.out::println, Throwable::printStackTrace);

bus.onNext(Notification.createOnNext(1));
bus.onNext(Notification.createOnError(new RuntimeException()));
bus.onNext(Notification.createOnNext(2));
bus.onNext(Notification.createOnNext(3));

You can use dematerialize to turn those notification back to regular calls to onXXX methods on your Subscriber yet have the original emitted still active.

15) defer expressed as flatMap
The operator defer let's you return a custom Publisher instance for each of the Subscribers. The effect can be simulated by using just with flatMap:
Func0<Observable<Integer>> supplier = () -> range(System.currentTimeMillis() & 1023, 5);

defer(supplier) == just("whatever").flatMap(v -> supplier.call());

In fact you could write it with concatMap as well:
defer(supplier) == just("whatever").concatMap(v -> supplier.call());

Even though using flatMap and concatMap this way looks like too much overhead, most modern libraries optimize away just with a special defer like operator that behaves like this:
Function<Integer, Observable<Integer>> function = v -> range(v, 2);

just("whatever").flatMap(function) == defer(() -> function.call("whatever"));

by exctacting the constant value from just and building a Func0 supplier that calls the original function with this constant and uses its returned Observable.
For this reason, using nest() with concatMap/flatMap is practically a no-op:
range(1, 5).nest().concatMap(o -> o.take(3)) == range(1, 5).take(3)

16) Emit elements of a list periodically
The source operator interval let's you create a periodic sequence of ever increasing Long values. Sometimes, we don't care about the values themselves and just want to act at the right time.
Then comes a requirement of emitting items from a list periodically, that is, with some fixed delay between elements. We can first map the Long values into the elements of the list, but we should also make sure we don't run out of indexes and get IndexOutOfBoundsException:
List<Integer> list = Arrays.asList(1, 2, 3, 4, 5);

interval(1, TimeUnit.SECONDS)
.takeUntil(t -> t == list.size() - 1)
.map(t -> list.get((int)t))
.subscribe(System.out::println)

Given the interval, we take its elements until the current running value is exactly the list's size minus one. The operator takeUntil executes the predicate after the value itself has been emitted downstream. In the map then, we simply call List.get() with a cast-down of the timer's value and now we get an item from the list.

16) Add a timed gap between elements
ConcatMap has the interesting property of preserving FIFO sequence. You can use that advantage to shift sequence items by a specific time or any arbitrary Publisher like interval:
range(1, 1_000)
.concatMap(t -> Mono.delay(1000).map( v -> t ))
.consume(System.out::println)

==

interval(1000)
.map( i )
.consume(System.out::println)

18)  Inner-join with flatMap
In SQL, we are used to write inner joins; the ones that pit one table against another table and forms all sorts of pairs. We can do such joins with flatMap:
Publisher<Integer> main = range(1, 5);
Publisher<String> slave = fromArray("a", "bb", "ccc");

main.flatMap(len -> slave.filter(s -> s.length() == len));

This will, for each main element stream through the slave Publisher and filtering out those elements of slave which have the current length from the main.
Some libraries offer overload that takes a two argument function which will receive these pairs and should produce some values out of them:
main.flatMap(v -> slave, (m, s) -> m == s.length());

Of course, one can return a Publisher from the second function and apply flatMap again to flatten out the resulting inner sequences, allowing us to "drop" irrelevant values from the final sequence:
main.flatMap(v -> slave, (m.s) -> m == s.length() ? just(s) : empty())
.flatMap(v -> v)

19) Eager concatenation
Sometimes, one would like to merge concurrently running sources and keep their order at the same time. The usual flatMap doesn't keep the order and the usual concatMap doesn't start the sources.
Many reactive libraries have an operator for this: concatMapEager (and concatEager) that gives the middle ground. In case your library doesn't support this operator, you can achieve a similar effect by concatenating pre-started sources via the help of an UnicastSubject:
range(1, 10)
.map(v -> {
    Observable<T> o = getSource(v);
    UnicastSubject<T> us = UnicastSubject.create();
    o.subscribe(us);
    return us;
})
.concatMap(v -> v)
.subscribe(System.out::println, Throwable::printStackTrace);

Here, values 1..10 are mapped to a source which is then started by subscribing an UnicastSubject to it. UnicastSubject caches values (unbounded) until a subscriber, such as that of inside concat subscribes to it and dutifully replays/relays items.
In case you don't have an UnicastSubject available, the are alternatives to it:

By using ReplaySubject instead, but note that it will retain all values until the whole sequence gets GC'd.

.map(v -> {
    Observable<T> o = getSource(v);
    ReplaySubject<T> rs = ReplaySubject.create();
    o.subscribe(rs);
    return rs;
})

By using replay() and connect()

.map(v -> {
    Observable<T> o = getSource(v);
    ConnectableObservable<T> co = o.replay();
    co.subscribe(e -> { }, Throwable::printStackTrace);
    co.connect();
    return co;
})

That extra subscribe is necessary because replay may not start actually running its source unless there are Subscriber(s) waiting for the items already.

By using replay() and autoConnect()

.map(v -> {
    Observable<T> o = getSource(v);
    Observable<T> co = o.replay().autoConnect();
    co.subscribe(e -> { }, Throwable::printStackTrace);
    return co;
})

Generally, there are two issues with these approaches:

They all run the sources to their completion instead of just prefetching certain amount. It requires a custom operator to solve.
By default, unsubscription won't unsubscribe the prestarted sources. You have to manually wire up the end Subscriber with all participants:

Subscriber<T> s = ...

// ...
.map(v -> {
    Observable<T> o = getSource(v);
    UnicastSubject<T> us = UnicastSubject.create();
    s.add(o.subscribe(us));
    return us;
})

// ...
.map(v -> {
    Observable<T> o = getSource(v);
    ConnectableObservable<T> co = o.replay();
    co.subscribe(e -> { }, Throwable::printStackTrace);
    s.add(co.connect());
    return co;
})

// ...
.map(v -> {
    Observable<T> o = getSource(v);
    Observable<T> co = o.replay().autoConnect(1, s::add);
    co.subscribe(e -> { }, Throwable::printStackTrace);
    return co;
})

20) defer via using
The operator defer let's you generate a source Observable/Publisher for each Subscriber whereas using let's you generate a resource per Subscriber, then use it to gereate a Observable/Publisher.
Therefore, you can imitate defer via a using setup where you create and ignore a resource and just create the source:
Supplier<Publisher<Integer>> s = () -> range(1, 10);

defer(s) => using(() -> "whatever", v -> s, v -> { });

However, you could also create the source itself as the resource and use identity-mapping on it:
using(s, v -> v, v -> { })

using(s::get, v -> v, v -> { })

The first shorter case is applicable if your library's using takes a Supplier and the second if it takes some () -> T generator function.
The reverse direction, namely expressing using as defer is a bit more involved:
Supplier<R> resource = ...
Function<R, Publisher<T>> source ==
Consumer<R> disposer;

defer(() ->
    fromCallable(resource)
    .flatMap(r -> {
        try {
            return source.apply(r)
            .doOnTerminate(() -> disposer.accept(r))
            .doOnUnsubscribe(() -> disposer.accept(r));
        } catch (Throwable ex) {
            disposer.accept(r);
            return error(ex);
        }
     })
);

In RxJava, a terminal event is followed by an unsubscribe call from most end-consumers (due to SafeSubscriber). In case the resource or the disposer is not idempotent, we have to make sure the dispose happens exactly once:
defer(() ->
    fromCallable(resource)
    .flatMap(r -> {
        AtomicBoolean once = new AtomicBoolean();
        try {
            return source.apply(r)
            .doOnTerminate(() -> disposeOnce(once, disposer, r))
            .doOnUnsubscribe(() -> disposeOnce(once, disposer, r));
        } catch (Throwable ex) {
            disposeOnce(once, disposer, r);
            return error(ex);
        }
     })
);

<R> void disposeOnce(AtomicBoolean once, Consumer<R> disposer, R resource) {
    if (once.compareAndSet(false, true)) {
        disposer.accept(resource);
    }
}

21) Caching and clearing
If we want to execute some code once and then hand out the generated values, such as login tokens or results of a network call, we usually can go for cache(), replay(), AsyncSubject, etc.
However, sometimes the data gets outdated and there is no way of clearing the structures above. But we can restart the whole process and make sure new subscribers get the fresh data if we cache the cache itself and use defer to get the current caching source:
final AtomicReference<Observable<Long>> cache = new AtomicReference<>(getSource());

public Observable<Long> getSource() {
    return Observable.fromCallable(System::currentTimeMillis).cache();
}

public Observable<Long> get() {
    return Observable.defer(() -> cache.get());
}

public void reset() {
    cache.set(getSource());
}

22) Compute a single value only when requested
By design, fromCallable executes the Callable immediately and doesn't wait for a downstream request to appear.
In case you want to compute only when requested, you can use the same jump-start trick with just().map() similar to Gem 10).
just("irrelevant")
.map(unused -> {
    try {
        return callable.call();
    } catch (Exception ex) {
        Exceptions.bubble(ex); // or Exceptions.propagate(ex)
    }
})
.subscribe(System.out::println, Throwable::printStackTrace);

Interesting. Do you mean Mono.fromCallable() (because when I looked at the source code it seemed to me that the Callable.call() only happens when there is a subscribe())? In general, what's a good way to assert or inspect statements like that? What did I misunderstand?

Here's the code from Gem 22 (with a concrete supplier and a log instead of using sysout):
        Callable<String> callable = () -> "foo";
        Mono.just("irrelevant").log().map(unused -> {
            try {
                return callable.call();
            } catch (Exception ex) {
                throw Exceptions.bubble(ex); // or Exceptions.propagate(ex)
            }
        }).subscribe(log::info, Throwable::printStackTrace);

and here's the output:
10:06:25.504 [main] INFO reactor.core.publisher.FluxLog -  onSubscribe(reactor.core.util.ScalarSubscription@5442a311)
10:06:25.506 [main] INFO reactor.core.publisher.FluxLog -  request(unbounded)
10:06:25.506 [main] INFO reactor.core.publisher.FluxLog -  onNext(irrelevant)
10:06:25.506 [main] INFO com.example.MonoFeaturesTests - foo
10:06:25.506 [main] INFO reactor.core.publisher.FluxLog -  onComplete()

The vanilla fromCallable()
        Mono.fromCallable(callable).log().subscribe(log::info, Throwable::printStackTrace);

Output:
10:05:13.317 [main] INFO reactor.core.publisher.FluxLog -  onSubscribe(reactor.core.subscriber.DeferredScalarSubscriber@1794d431)
10:05:13.318 [main] INFO reactor.core.publisher.FluxLog -  request(unbounded)
10:05:13.318 [main] INFO reactor.core.publisher.FluxLog -  onNext(foo)
10:05:13.318 [main] INFO com.example.MonoFeaturesTests - foo
10:05:13.318 [main] INFO reactor.core.publisher.FluxLog -  onComplete()

What's the difference?

Please don't add non-gem comments to this issue but use #23 instead.


Display currently running agent jobs for all monitored instances

Discussed in https://github.com/trimble-oss/dba-dash/discussions/708

Originally posted by tdinh2011 September 23, 2023
It would be helpful to have a view that shows currently running jobs for all monitored instances. This would help quickly identify long-running overnight jobs.

It would be useful to see what long-running jobs are currently running.  The DBA Dash GUI doesn't have any direct connectivity to the monitored instances so it would rely on reporting the data from the last snapshot.  It would be reasonable to collect this every 1min as a default.  As the data isn't live, some jobs reported as running might already have completed.  The real value is being able to identify the long-running jobs that are still in progress.
In some cases, you will be able to see in the running queries snapshot that a job is still running.  This won't work if the job isn't doing any data access though.  Also, you can't easily see what jobs are still running across all your SQL instances.  Capturing jobs currently running would still be useful.
It would be useful to see what jobs are running across all servers at root level.  It would also be useful at the server level.  It could maybe be shown on the timeline.
Maybe the job execution data could also be linked to the running queries and slow query data to make troubleshooting agent job performance easier.

Included in 2.48.0 🎉
https://dbadash.com/blog/whats-new-in-2.48.0/


JwtSecurityTokenHandler can't be used with WsFederation middleware

Looks like a regression of #18.
The 5.x version of JwtSecurityTokenHandler has only a CanReadToken( string token ) and thus rejects things like <wsse:BinarySecurityToken ValueType="urn:ietf:params:oauth:token-type:jwt"... when used with the WsFederationMiddleware in Katana 4.0.
Previous version (when it lived in System.IdentityModel.Tokens.JwtSecurityTokenHandler) also had a CanReadToken( XmlReader reader )

Oh wow, seems at this point that nothing is going to happen about this issue.

@solidstore @MauroMS this work is stacked behind a public release of 6.4.2 where we have made several changes that were used and tested inside microsoft.
Expect this work in an up-coming 6.4.2+ version.

This is will help users that want to use JWT's and WCF or other SOAP messaging.

@solidstore unfortunately we are unable to support this.


terraform-ls: 0.7.0 -> 0.8.0, switch to buildGoModule

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[x] macOS
[x] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 102563 1

1 package built:

terraform-ls

Result of nixpkgs-review pr 102563 1

1 package failed to build:
- terraform-ls

$ nix build --no-link --keep-going --option build-use-sandbox relaxed -f /home/bhipple/.cache/nixpkgs-review/pr-102563/build.nix
builder for '/nix/store/n61ca2204r8vbi0xmmmsdvx7r90200q0-terraform-ls-0.8.0.drv' failed with exit code 1; last 10 log lines:
  ok       github.com/hashicorp/terraform-ls/internal/terraform/addrs      0.003s
  === RUN   TestExec_timeout
      exec_test.go:71: error fetching checksums: Get "https://releases.hashicorp.com/terraform/0.13.1/terraform_0.13.1_SHA256SUMS": dial tcp: lookup releases.hashicorp.com on [::1]:53: read udp [::1]:43802->[::1]:53: read: connection refused
  --- FAIL: TestExec_timeout (0.00s)
  === RUN   TestExec_cancel
      exec_test.go:71: error fetching checksums: Get "https://releases.hashicorp.com/terraform/0.13.1/terraform_0.13.1_SHA256SUMS": dial tcp: lookup releases.hashicorp.com on [::1]:53: read udp [::1]:55281->[::1]:53: read: connection refused
  --- FAIL: TestExec_cancel (0.00s)
  FAIL
  FAIL      github.com/hashicorp/terraform-ls/internal/terraform/exec       0.008s
  FAIL
cannot build derivation '/nix/store/acnc7gzrip1ffzyjwb2lk89rzriblxiy-env.drv': 1 dependencies couldn't be built
[2 built (1 failed), 0.0 MiB DL]

hmm, I've reproduced it with --check. I suppose I just make doCheck = false to prevent downloads made by tests in sandbox mode.

Result of nixpkgs-review pr 102563 run on x86_64-darwin 1

1 package built:

terraform-ls

This pull request has been mentioned on NixOS Discourse.


Question about some training params

Hello,
Thank you for the repository.
Please, I have some questions in order to reproduce your results.
1- Which training parameters and memory size did you use for imagenet100? In the paper, you just provide those of cifar100 and imagenet1000.
2- Can you please provide me with the image list with the class order used for cifar100 experiments?
3- Does an initial resizing of imagenet's images affects training? Did you use the original image size? (I'm not talking about the resized crop in the code, but about resizing images and storing them on the disk before training the models).
4- Do you have a complete set of results of your baseline? (Ft+distillation, before bias removal layer)
Thank you for your help

@EdenBelouadah Thanks for the interest in our work. Nice talking with you again:)

For imagenet-100, training parameters are the same with imagenet-1000 in section 6.2 of the paper. Memory size is 2000 ( 20 samples each class for a total 100 classes, 20*100=2000).
The class order of cifar is the same with icarl, which is random generated using the random seed 1993 (https://github.com/srebuffi/iCaRL/blob/master/iCaRL-TheanoLasagne/main_cifar_100_theano.py#L28).
We use the original image size, which are the originally downloaded ones. I don't know if some initial resizing will affect the training. The only processing we used is the vgg processing(https://github.com/wuyuebupt/LargeScaleIncrementalLearning/blob/master/vgg_preprocessing.py).
I don't think that I have those results. Sorry about that. But results from "before bias removal layer" can be found through the code output logs (should be the values of "accuracy_before").

@wuyuebupt , nice to talk to you too and thank you for your fast reply :)
1- Do you reduce the number of images per class when incrementing? or 20 is freezed over all states?
2-3-4- Thank you !

@EdenBelouadah You are welcome.

Yes, the examplers for each class are reduced when incrementing, e.g. https://github.com/wuyuebupt/LargeScaleIncrementalLearning/blob/master/imagenet_main.py#L440 and https://github.com/wuyuebupt/LargeScaleIncrementalLearning/blob/master/imagenet_main.py#L489

Yes, thank you, I just wanted to make sure because it was not so clear for me from your explanation.
Thanks a lot!


SSL - read operation timed out

For some months now I have been getting errors sometimes when the cron job grabs helpdesk messages from the IMAP queue.
It doesn't cause too many problems - the emails will usually be successfully grabbed 5 minutes later, and will always run fine when run from an interactive shell like this:
>python manage.py get_email
I suspect it relates to a timeout issue.
I wonder if get_email.py, line 207, should be wrapped in a try / catch?
Or ... maybe introduce a timeout value in helpdesk's settings which defaults to the current value?
Does anyone else get these?
If i ran the cron job like this:
python manage.py get_email --quiet >/dev/null
or similar I would never have noticed this happening.
Traceback (most recent call last):   File "manage.py", line 10, in <module>     execute_from_command_line(sys.argv)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line     utility.execute()   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute     self.fetch_command(subcommand).run_from_argv(self.argv)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv     self.execute(*args, **cmd_options)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute     output = self.handle(*args, **options)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/helpdesk/management/commands/get_email.py", line 73, in handle     process_email(quiet=quiet)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/helpdesk/management/commands/get_email.py", line 106, in process_email     process_queue(q, logger=logger)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/helpdesk/management/commands/get_email.py", line 207, in process_queue     status, data = server.fetch(num, '(RFC822)')   File "/usr/local/lib/python2.7/imaplib.py", line 456, in fetch     typ, dat = self._simple_command(name, message_set, message_parts)   File "/usr/local/lib/python2.7/imaplib.py", line 1088, in _simple_command     return self._command_complete(name, self._command(name, *args))   File "/usr/local/lib/python2.7/imaplib.py", line 910, in _command_complete     typ, data = self._get_tagged_response(tag)   File "/usr/local/lib/python2.7/imaplib.py", line 1017, in _get_tagged_response     self._get_response()   File "/usr/local/lib/python2.7/imaplib.py", line 929, in _get_response     resp = self._get_line()   File "/usr/local/lib/python2.7/imaplib.py", line 1027, in _get_line     line = self.readline()   File "/usr/local/lib/python2.7/imaplib.py", line 1189, in readline     return self.file.readline()   File "/usr/local/lib/python2.7/socket.py", line 451, in readline     data = self._sock.recv(self._rbufsize)   File "/usr/local/lib/python2.7/ssl.py", line 734, in recv     return self.read(buflen)   File "/usr/local/lib/python2.7/ssl.py", line 621, in read     v = self._sslobj.read(len or 1024) ssl.SSLError: ('The read operation timed out',)

Also we get this error (less frequently, but i'm sure it's related)
Traceback (most recent call last):   File "manage.py", line 10, in <module>     execute_from_command_line(sys.argv)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line     utility.execute()   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute     self.fetch_command(subcommand).run_from_argv(self.argv)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv     self.execute(*args, **cmd_options)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute     output = self.handle(*args, **options)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/helpdesk/management/commands/get_email.py", line 73, in handle     process_email(quiet=quiet)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/helpdesk/management/commands/get_email.py", line 106, in process_email     process_queue(q, logger=logger)   File "/home/e-smith/files/ibays/django/django_app/m10_repo/m10virtualenv/lib/python2.7/site-packages/helpdesk/management/commands/get_email.py", line 182, in process_queue     int(q.email_box_port))   File "/usr/local/lib/python2.7/imaplib.py", line 1166, in __init__     IMAP4.__init__(self, host, port)   File "/usr/local/lib/python2.7/imaplib.py", line 173, in __init__     self.open(host, port)   File "/usr/local/lib/python2.7/imaplib.py", line 1178, in open     self.sslobj = ssl.wrap_socket(self.sock, self.keyfile, self.certfile)   File "/usr/local/lib/python2.7/ssl.py", line 911, in wrap_socket     ciphers=ciphers)   File "/usr/local/lib/python2.7/ssl.py", line 579, in __init__     self.do_handshake()   File "/usr/local/lib/python2.7/ssl.py", line 808, in do_handshake     self._sslobj.do_handshake() ssl.SSLError: ('_ssl.c:574: The handshake operation timed out',)

I added a commit to master that wrapped the IMAP login attempt in a try-except block. I simply log that it failed (logging a timeout if we see the ssl.SSLError in your debug stuff above) and exit the script, so it closes instead of crashes. I figured this is enough for now since cron will simply run it again at whatever your next interval is, and the error is logged. If you have some time, please check out this update from master. I'll wrap this into the next bug fix release very soon. Thanks!

I added this to the 0.2.3 release, so I'm going to go ahead and close it for now. But please re-open or file a new ticket if you spot any problems or have further suggestions. Thanks!


[Bug?] Timer resets when switching tasks while it's actively counting down

Steps to Reproduce:

Start the timer for a specific task.
While the timer is counting down, click on another task
Timer resets

Not sure what the expected behavior is. Pomofocus.io continues the countdown after a popup asking if we want to switch tasks.

Yes that's intended honestly.
But sure I can confirm with the user before they switch more.
Thanks for contributing

I'd love to give it a shot! @teoh4770

@alexsan9051 assigned.
Remember to fork it and make pr after you finish it.
Let me know if you need any help.

Reason we close this issue
"So I just saw this post about the reason we should against functions like alert and confirm. Basically we want the user to have fully control over the app itself, we don't want to promptly stop the user from using the app just to ask for confirmation(this is where I dislike about the original pomodoro app), unless (in my opinion) it is a destructive operation like delete a file permanently. "


Update contributors (I should have done this in #18) and bump pypi version

Hola @vsudilov! You rewrote the whole code base (for the better) and deserve more credit. Would you be willing to merge this first before I merge https://github.com/andycasey/ads/pull/27?

Cheers!


Snapshots are different on linux and windows when contain \\

🐛 Bug Report
Snapshoting RegExp on windows and linux are difference when it is contains \\.
To Reproduce
describe("regexp", () => {
	it("should work", () => {
		expect({ test: /[\\/]node_modules[\\/]/ }).toMatchInlineSnapshot();
	});
});

Expected behavior
Snapshots should escaped property and equally on windows and linux
Link to repl or repo (highly encouraged)
Original issue: https://github.com/webpack/webpack/pull/10443/
envinfo

npx: installed 1 in 2.375s

  System:
    OS: Linux 5.3 Ubuntu 19.10 (Eoan Ermine)
    CPU: (6) x64 AMD FX(tm)-6300 Six-Core Processor
  Binaries:
    Node: 10.15.2 - /usr/bin/node
    Yarn: 1.13.0 - /usr/bin/yarn
    npm: 6.13.7 - /usr/local/bin/npm
  npmPackages:
    jest: ^25.1.0 => 25.1.0

Additional information: there were no problems using jest@24, looks like it is a regression

@pedrottimark ideas?

The regexp part of pretty-format hasn't changed for years
https://github.com/facebook/jest/blob/003bd50afaf0022381ffa9eb30abd511fad2ed2e/packages/pretty-format/src/index.ts#L157-L163

So I'm not sure where to begin looking into this... Maybe the equality function just lies? It was tweaked in #9167. RegExp equality went from
    return (
        a.source == b.source &&
        a.global == b.global &&
        a.multiline == b.multiline &&
        a.ignoreCase == b.ignoreCase
      );

to
return a.source === b.source && a.flags === b.flags;

I wouldn't expect that to result in this change, though...
@jeysal ideas?

@SimenB I can spent my time on searching a problem, I'm afraid that it will take me more time for me, so I created the problem, suddenly there are some known potential problem areas

Hmm I'd be surprised if it was related to that too. It should only matter for flags other than those 3


add "sign in to cloud" button to landing page

With this commit I added a "sign in to cloud" button to the landing page. It depicts a rocket on the button. This PR addresses #40.
Sadly the link to the cloud is quite long - but I guess there is nothing we can do about that?
Checks for reviewers

[x] Is the button displayed properly?
[x]  Does the link work?
[x] Can the link be shortened

great :) All done!


Send a "memory_policy" update with addNewVm and editVm API calls

Resolves issue #499

This change is

@mareklibra I have verified that this PR fixes the issue.


Expander card keeps refreshing when adding cards

Symptom:
Card refreshes when trying to add information
Steps to reproduce:
Add card > Expander card

Add a title card (e.g. Entity card)

Try to modify information in the Card Configuration window = Card Configuration window refreshes itself and removes entries.  I cannot add any entries as it keeps refreshing every couple of seconds.

OS info:
Home Assistant 2023.8.4 Supervisor 2023.08.1 Operating System 10.5 Frontend 20230802.1 - latest
Expander card info:
0.0.4 version
Please let me know if there is further information needed to help resolve the issue I'm experiencing.
Thanks.

I also noticed if you switch to another card it will overwrite it.

Same symptom here. Only present when title card has been added and in visual editor mode. Symptom not present if title card added and switching to code editor mode.

same issue appears on my end

Same here

Same :/

Same problem here.
Adding/removing cards is no longer possible, as it keeps refreshing and going back to the initial view.
Only way to do it now is with YAML.
@Alia5

Same here. Makes it hard to work with the card, unfortunately.

Yea, I would love to use this, but it just overwrote half my dashboard with this weird refresh behavior, so its getting uninstalled.

Same, its legit making the card unusable I can't add any cards to it. Smh.

Same error, it's okay until I try to add a title card then it becomes unusable.
Such a shame as its a great card!

Same here.

Still the  same, highly unstable


HTTPS_PROXY / HTTP_PROXY?

Hello
We are behind an outgoing proxy, and I suspect Mattermost not being able to connect to stuff
For example, docker logs mattermost :
{"timestamp":"2021-12-15 23:58:50.950 Z","level":"debug","msg":"ERROR: sending request - Post \"https://pdat.matterlytics.com/v1/batch\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","caller":"log/log.go:184","source":"rudder"}

I remember with a previous version of Mattermost (https://github.com/mattermost/mattermost-docker), I had to somehow configure outgoing http proxy. I precisely modified the Dockerfile to recompile it.
Please, what's the solution here?
Thanks!

@Bibelo the solution to what? The error message you've posted isn't fatal at all. Is every outgoing request failing? If so you need to configure this inside the container or on the docker host.

@mrckndt Thank you for your answer and sorry for my late reply.
There's no need to ask if every outgoing request is failing, since by design, every request cannot go out. Configuring an outgoing proxy server is necessary, always have been in my organization, and have been in the previous version of MM I'm currently using. It's a standard feature you find in products.
My question then was how to add outgoing http/https proxy to the image. It's not as simple as exporting a variable.
It should be somewhere, either in the MM config files or as a docker variable in the docker-compose. I found no documentation about it, that's why I'm asking.
Thanks in advance for your help.

I have the same problem. Outgoing connections don't get through because the proxy has to be used for that (in my case).

@Bibelo The documentation you're searching for is here. This needs to be applied to the proxy facing container (either Nginx or Mattermost). It should also work when a proxy is configured on the Docker host.
@bor8 a bit more information would be great but the above is true for you too if using an outgoing proxy.

Thanks, it worked, this error Received invalid response from the server. is gone!


Avoid race condition on deploying lb with libvirt

Why is this PR needed?
Fixes an issue that can arise when manually deploying the terraform templates with libvirt
Fixes #
N/A
What does this PR do?
There seems to be a race condition on the 2 resources
of the libvirt lb (reboot, haproxy) which are launched
inmediatly after the cloud init resource has finished
and can lead to the haproxy resource never completing
because the machine rebooted.
Instead, make the reboot dependant on the haproxy
resource, so its done in a linear fashion.
Anything else a reviewer needs to know?
N/A
Info for QA
N/A
Related info
N/A
Status BEFORE applying the patch
Creating libvirt resources with the terraform templates could lead to a timeout of the lb_push_haproxy_cfg resource
Status AFTER applying the patch
Creating libvirt resources with the terraform template will never timeout the lb_push_haproxy_cfg resource
Docs
N/A
Merge restrictions
(Please do not edit this)
We are in v4-maintenance phase, so we will restrict what can be merged to prevent unexpected surprises:
What can be merged (merge criteria):
    2 approvals:
        1 developer: code is fine
        1 QA: QA is fine
    there is a PR for updating documentation (or a statement that this is not needed)

oh, this seems to be fixed in a different manner here: https://github.com/SUSE/skuba/pull/758


KAFKA-15720: KRaft support in DeleteTopicTest

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Hi @dengziming , can you please take a look at this PR whenever you have time? There are a few test cases that I couldn't run in KRaft mode. They require replica states from controllerContext of KafkaServer. Some of them also require zookeeper's path to see if topics are marked as deleted but not yet deleted.  I left those ones as they are, because they are more zookeeper specific scenarios. I was wondering what you think, or if you could suggest a way to enable KRaft for these test cases. Thank you

Hello @tinaselenge , there are some flaky related to this, we should take some time to investigate it, and we should trigger the CI more times to ensure less flaky.
https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14846/2//#showFailuresLink
Build / JDK 17 and Scala 2.13 / kafka.admin.DeleteTopicTest.testIncreasePartitionCountDuringDeleteTopic()
Build / JDK 17 and Scala 2.13 / kafka.admin.DeleteTopicTest.testDisableDeleteTopic(String).quorum=kraft
Build / JDK 17 and Scala 2.13 / kafka.admin.DeleteTopicTest.executionError

Thanks @dengziming. I will investigate them.

@dengziming I believe I fixed the flaky tests which did not appear to be failing in any of the recent builds. The builds still failed but due to the non related tests.

Hello @tinaselenge thanks for the update, here are some of my suggestions:

If some controllerContext stuff is required, such as testIncreasePartitionCountDuringDeleteTopic, it's testing some concurrent problems, we use a single event queue in KRaft mode so it's unnecessary to validate these states, and we can put these validations in a if (!kraftTest()) clause.
If we need to validate zk path, we can also put them in if (!kraftTest()) clause.
For some operations using adminZkClient we can use AdminClient to replace them, use AdminClient.deleteTopic to replace adminZkClient.deleteTopic.

and please merge trunk to trigger CI for at lease 3 times, we have been deeply affected by flaky tests in the past and we should avoid it as possible.

@dengziming I have triggered the build 3 more times and DeleteTopicTest passed. I have also enabled KRaft for testIncreasePartitionCountDuringDeleteTopic (they didn't have much common code so simply separated by a single if (!kraftTest()) clause. I also added comments for 2 other tests explaining why KRaft is not enabled.

@dengziming thank you again for reviewing the code. I addressed your comments, please let me know if I missed anything or have any further comments.

Hi @tinaselenge and @dengziming
I think this change broke the build. Right now Kafka can't be compiled with Scala 2.12, see the error here: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14846/15/pipeline/9

[Error] /home/jenkins/workspace/Kafka_kafka-pr_PR-14846/core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala:490:62: the result type of an implicit conversion must be more specific than Object

Fix is here: https://github.com/apache/kafka/pull/15290

Thank you @jlprat for the fix!


Update/upgrade for component axial linkage

Description
The current methodology for determining component axial linkage was found to be error prone. The current approach requires that Components be the same class type and have an area with an absolute difference of 10^-3. For some cases, this can provide false positive linkage. To improve this, an overhaul of the methodology was undertaken. In this PR, this methodology is updated/upgraded to to account for the following: 1) Component classes must be solids, 2) the same class type, 3) have the same multiplicity, 4) and meet overlap criteria. A runLog.warning has also been added if a Component is found to be linked to multiple Components in a neighboring block.
The methodology can be found in axialExpansionChanger.py::AssemblyAxialLinkage::_determineAxialLinkage. Several relevant unit tests have been added to cover various expected cases - see test_AxialExpansionChanger.py::TestLinkage. Other unit tests have also been added to maintain coverage in basicShapes.py and complexShapes.py.

Checklist

[x] The code is understandable and maintainable to people beyond the author.
[x] Tests have been added/updated to verify that the new or changed code works.
[x] There is no commented out code in this PR.
[x] The commit message follows good practices.
[x] All docstrings are still up-to-date with these changes.

@john-science @onufer


fix to_compressed_netcdf when coords are chunked

ds_bitrounded.to_compressed_netcdf() didn't work when data is eager but coords are lazy.
workaround:
for c in ['clon','clat','elon','elat']:
    ds_bitrounded.coords[c] = ds_bitrounded.coords[c].compute()

ds_bitrounded.to_compressed_netcdf() didn't work when data is eager but coords are lazy.
workaround:
for c in ['clon','clat','elon','elat']:
    ds_bitrounded.coords[c] = ds_bitrounded.coords[c].compute()

This looks super strange to me if these are actual coordinates. I thought, xarray always loads coordinates eagerly because it could otherwise not do the selection by coordinates, which can actually be a trouble when you load for example a very large time series where the coordinates do not fit into the memory. The PR looks decent though.

These are not dimension coordinates but multidim coords. Therefore the can be lazy.


Touch zoom option?

keyboardZoom and mouseZoom can be used to enable/disable zooming using the keyboard or the mouse.
Is there also an option for touch/pinch style zooming? How could zooming be completely disabled?

Set the minHfov and maxHfov parameters to the same value.


display the active probes list in status api

display the activated probes (without error) for agent and analyzer in status API.
in case if any of the given probes is not active due to error the user needs to go through the logs.

https://github.com/skydive-project/skydive/pull/756


Create dataset loader for SREDFM

Dataloader name: sredfm/sredfm.py
DataCatalogue: http://seacrowd.github.io/seacrowd-catalogue/card.html?sredfm

Dataset
sredfm

Description
SREDFM is an automatically annotated dataset for relation extraction task covering 18 languages, 400 relation types, 13 entity types, totaling more than 40 million triplet instances. SREDFM includes Vietnamnese.

Subsets
SREDFM_vi

Languages
vie

License
Creative Commons Attribution Share Alike 4.0 (cc-by-sa-4.0)

Homepage
https://github.com/babelscape/rebel

HF URL
https://huggingface.co/datasets/Babelscape/SREDFM

Paper URL
https://aclanthology.org/2023.acl-long.237/

This is interesting, I'll take this first and see whether this can be done under a week, else I might release the task to others.
Btw, @SamuelCahyawijaya do we have a increasing bonus system after some time if the issue hasn't been picked up by anyone?

#self-assign

I'll take this once #62 has been reviewed. Will start it by next week.

Will try to get this done by EoW

Ugh, didn't have the chance to do this. will release this and see if anyone else can take this instead.

#self-assign

Hi, I did worked on this. The dataloader works, but apparently the test are hard to get passed, especially the seacrowd schema one. IIRC, I was having issues with IDs (duplicate). Currently have no time yet to fix it.

Currently discussing it with @SamuelCahyawijaya

Basically I got two problems

Spacing problem (the text and text offset asserted not equal. The content is the same, but the spacing is different)

Id uniqueness problem

For second problem, I am still not sure why because:

For source schema, I assume it only checks the yielded id (the yield ..., { ... } at the example loop)
So for ID uniqueness, it is most likely coming from the seacrowd shema, BUT assuming that this is the ids the test check
"id": example["docid"], --> skip the same doc id
"passages": passages, --> use custom id (counter)
"entities": entities, --> counter
"relations": relations, --> counter

Can look at the current full code here

I think @SamuelCahyawijaya is currently busy. Might need another help.

For 1, I believe the text_offset was generated from the text field, but I saw on the offset text that a new line isn't present in the original text. Is it expected?
For 2, when I rechecked the schema, it checked the IDs defined on the example level to ensure they were unique. In your implementation, you're defining the entity & relation ID only by specifying an iteration counter, which leads to duplication on the check. Perhaps the workaround will be similar to what you did on the passage ID, appending it to some unique identifier for distinguishing entity ID and relation ID.

OK, will take a look into this again in near time.

For number 2) it is resolved now. I think I misunderstood the concept id at first.
Now, for problem number 1) -> yes, the problem is in the newline. Content-wise is the same

omg, I passed the tests 😂
The problem actually lies in the real dataset. The entities from the source datasets have "tidy" whitespaces (they don't have newlines). If you think about it, if the real passages are messy, the entity taken from it should also be messy.
That being said, I suspect the source dataset does not actually take the entity from the passage cause the passage is a bit chaotic (in terms of whitespaces).

 omg, I passed the tests 😂
The problem actually lies in the real dataset. The entities from the source datasets have "tidy" whitespaces (they don't have newlines). If you think about it, if the real passages are messy, the entity taken from it should also be messy.
That being said, I suspect the source dataset does not actually take the entity from the passage cause the passage is a bit chaotic (in terms of whitespaces).

Awesome, hahaha. Glad to hear that you've resolved the issue! 👍


[Wise] Editing loss = nan + 0

I run the sample code of WISE and use the same ZSRE dataset file. It turns out that the loss is nan and the edit fail. Is there anything I misunderstand or I skip some step?
This is the code I use,

import sys
import os
import json
# Add the parent directory to the sys.path
from EasyEdit.easyeditor import WISEHyperParams, BaseEditor
from huggingface_hub import login

K = 1000
edit_data = json.load(open('./data/zsre_mend_edit.json', 'r', encoding='utf-8'))[:K]
loc_data = json.load(open('./data/zsre_mend_train.json', 'r', encoding='utf-8'))[:K]
loc_prompts = [edit_data_['loc'] + ' ' + edit_data_['loc_ans'] for edit_data_ in loc_data]

prompts = [edit_data_['src'] for edit_data_ in edit_data]
rephrase_prompts = [edit_data_['rephrase'] for edit_data_ in edit_data]
target_new = [edit_data_['alt'] for edit_data_ in edit_data]
locality_prompts = [edit_data_['loc'] for edit_data_ in edit_data]
locality_ans = [edit_data_['loc_ans'] for edit_data_ in edit_data]
locality_inputs = {
    'neighborhood':{
        'prompt': locality_prompts,
        'ground_truth': locality_ans
    },
}
hparams = WISEHyperParams.from_hparams('./EasyEdit/hparams/WISE/llama-7b.yaml')

hparams.model_name = "TheBloke/Llama-2-7B-fp16"
editor = BaseEditor.from_hparams(hparams)
metrics, edited_model, _ = editor.edit(
    prompts=prompts,
    rephrase_prompts=rephrase_prompts,
    target_new=target_new,
    loc_prompts=loc_prompts,
    locality_inputs=locality_inputs,
    sequential_edit=True
)
# a =

It's possible that the issue you're encountering with the loss becoming nan is due to fp16 overflow. Have you experimented with editing "TheBloke/Llama-2-7B-fp16" using any other editing methods? Do you have the GPU resources to try using fp32 instead?

hi, have you solved your issue yet?


[BEAM-2068] Pin apitools to 0.5.8

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @aaltay PTAL

Could you add a JIRA issue to upgrade to 0.5.9 and as a warning for what is wrong.

R: @aaltay done.

Should we actually pin all dependencies? We make the assumption that our dependencies would not introduce breaking changes without a major version change but it happens frequently.

I don't have strong opinions around this but here are some of my thoughts:

We can upgrade at our own pace and can always promise that beam head is stable.

Having really strict version pinning makes it hard for users of Beam to always make sure they are using the same versions of the library. So if a user was already using a higher version than us for a library that made safe changes, it'll be annoying for them but I guess if we generally stay up to date with latest versions very few ppl will have that issue.
Might be worthwhile exploring some sort of version-eye style system to make sure we're upgrading on time.

Your have valid points. Then at release time we need to ensure that corresponding pieces (e.q. dataflow container) has at least the minimum version of all referenced dependencies.

LGTM

Thanks


Routing doesn't work as expected

Looks like 17.0.0 release broke routing.
This code doesn't work as expected any more: https://github.com/Novage/wt-tracker/blob/b6c34fb3d16f63dd12bed6be74050b447aceb3c0/lib/run-uws-tracker.ts#L89
I want to server index "/", "/stats.json" and 404 for all other requests "/*".
But when I add "/*" route all other routes stop working.

I can't reproduce with this:
uWS.App().get('/', (res, req) => {
  res.end('Index');
}).get('/status.json', (res, req) => {
  res.end('Status');
}).any('/*', (res, req) => {
  res.end('Anything');
}).listen(port, (listenSocket) => {
	console.log(listenSocket);
});

Matching rules have changed in 17, you match not by order but by how specific your route is (most specific first).

If you can provide a minimal example where the issue happens I can look at it, but this new router was tested with unit tests more than any other part of the code, so it should work properly.
I can't see the issue.

This also works:
for (let i = 0; i < 3; i++) {
	uWS.App().get('/', (res, req) => {
	  res.end('Index from app ' + i);
	}).get('/status.json', (res, req) => {
	  res.end('Status from app ' + i);
	}).any('/*', (res, req) => {
	  res.end('Anything from app ' + i);
	}).listen(3000 + i, (listenSocket) => {
		console.log(listenSocket);
	});
}

Ok. One more call is missing. So the code that worked before but doesn't work now:
require("uWebSockets.js").App()
.ws("/*", {})
.get('/', (res, req) => {
    res.end('Index');
})
.get('/status.json', (res, req) => {
    res.end('Status');
})
.any('/*', (res, req) => {
    res.end('Anything');
})
.listen(8080, (listenSocket) => {
    console.log(listenSocket);
});

I am not sure if it is intended, but it responds "Anything" on "/" and "/status.json" requests.


fix(ethers-etherscan): Broken Etherscan URL Construction

Motivation
The ethers-etherscan crate constructs string urls with an additional slash, creating an invalid URL. For example, when running a forge verify-contract, the generated etherscan url contains an extra slash.
Example 1

Example 2

Solution
Remove the additional slash in the url construction functions returning strings.
PR Checklist

[x] Added Tests

❌ (NOT NEEDED) Added Documentation
❌ (ethers-etherscan not included in changelog ?) Updated the changelog

needs fmt @abigger87 lgtm otherwise


Modify Acceptance Criteria Infrastructure to Summarize Assignment of On-board Survey

What existing problem does the pull request solve and why should we include it?
The existing Acceptance Criteria infrastructure include numerous methods for summarizing observed and simulated data. Here, minor adjustments are made to facilitate the comparison of outcomes from the assignment of the on-board survey data to observed data.
What is the testing plan?
Make sure core Acceptance Criteria summaries are not disturbed by the changes.
Demonstrate the code is solid by discussing how results are verified and covered by tests

[ ] Code for this PR is covered in tests
[ ] Code passes all existing tests

Code formatting
Code should be PEP8 compliant before merging by running a package like black

[ ] Code linted

Applicable Issues
None

@lmz, @gregerhardt, @FlaviaTsang
Per our call today, here are some distance summaries. Note two caveats: (1) we are using the midday roadway distance skim and (2) the skims are composite paths, so the walk trips are those with no probability of taking transit (there are others for which walking is part of the composite path).

For all observed movements (weighted)

count     31,765
trips    357,185
mean         11.14
std          11.41
min           0.187
25%           4.48
50%          10.58
75%          21.15
max          84.93

For movements with a transit path (weighted)

count     27,864
trips    291,011
mean         13.24
std          11.66
min           0.82
25%           6.13
50%          12.56
75%          23.32
max          84.93

For movements with a walk-all-the-way path, but no transit path (weighted)

count     3,901
trips    66,174
mean         1.90
std          0.82
min          0.18
25%          1.41
50%          1.91
75%          2.49
max         10.45

And then the same three summaries for observed trips that start and end in SF.

All observed intra-SF movements (weighted)

count      9,360
trips    152,785
mean          3.98
std           2.25
min           0.18
25%           2.44
50%           3.94
75%           5.96
max          12.88

Intra-SF movements with a transit path (weighted)

count      6,872
trips    106,640
mean          4.95
std           1.99
min           0.82
25%           3.58
50%           4.90
75%           6.64
max          12.88

Intra-SF movements with a walk-all-the-way path, but no transit path (weighted)

count     2,488
trips    46,144
mean         1.73
std          0.63
min          0.18
25%          1.30
50%          1.76
75%          2.24
max          4.20

To me, this strengthens the case to force a transit boarding, as the 25th percentile walk distance is greater than one mile.


Add resource navigation buttons  #293

Fixes #293

Also, the logic of the code is broken somehow.
If I use the buttons in a given section, "Licenses" for examples, everything works fine, but if you start going to other section, the prev/next is acting weird.

Also, the logic of the code is broken somehow. If I use the buttons in a given section, "Licenses" for examples, everything works fine, but if you start going to other section, the prev/next is acting weird.

Yes @tdruez, got it fixed in the https://github.com/nexB/scancode.io/pull/469/commits/cd674fbec747d342fc7c555e8372deff1489d444 commit

Thanks for your contribution @lf32 !

Thanks for the merge @tdruez!


Hooks (useState) invalidation

Replacing the declaration w with the comment in the next line doesn't reset the state.
export function App() {
  const [v] = useState(Math.random());

  const [w] = useState(Math.random());
  // const [w] = useState(Math.random() + 1);

  return (

      App

      {v}
 {w}

  );
}

This might seem fine but if the datatype of the state changes and the value is used accordingly, you might get an invalid UI state.
(React Fast Refresh invalidates App's state in this case and also when renaming the variable name w to something else for the same reason.)

Just tested this locally and it works for me, is it possible you're not adding react-refresh/babel to your babel plugins? This was implemented as of @prefresh/core version 0.5.0

is it possible you're not adding react-refresh/babel to your babel plugins

Indeed 😬 👍

Does @prefresh/vite v0.10.0 need react-refresh/babel installed?  I am getting broken return type behavior on initial render passes for preact/compat aliased hooks.
https://github.com/travomic/battle-orb/pull/24/files#diff-d47481d24720bd5625e4e96253f20f6ad872730b8c8b3871a91c873a836b9acaR12


Any example for Object Detection through NAS is availble?

Describe the issue:
Could you provide an example for Object Detection through NAS?

@xmyyzy123 thanks for raising this issue. We believe the current NAS example is sufficient for users to learn how to build search space with NNI.
As you are seeking for an example for object detection, I guess you want to have a reasonably good search space for object detection in this example, which is another story, i.e., what is a good search space for object detection.
If you have a search space in mind but do not know how to program it with NNI, feel free to tell us your search space, we can figure out how to program it.
BTW, we encourage external contributors to contribute more examples:)

@QuanluZhang  thanks for your reply😄. I look through samples about nas, finding that current datasets are about image classification(i.e., CIFAR10). How do I use COCO dataset(object detection dataset) for searching?  Could you provide an example?🤗


Is it okay to use arduino mega 2560?

I will use rplidar A1, arduino mega 2560 and raspiberry 3B. Is it okay to use your code???

I do not own "arduino mega 2560", so I can not answer accurate things.
However, I think that there is no difference other than arduino's buffer size.
Please adjust the size according to your environment "11. Remodel the rosserial library. Execute below." procedure.

Is this code to adjust the buffer size???
typedef NodeHandle_<ArduinoHardware, 15, 15, 128, 1024, FlashReadOutBuffer_> NodeHandle;
If it’s right, what does this code mean? Can you explain?

Is this code to adjust the buffer size???

Yes. That's right.

If it’s right, what does this code mean? Can you explain?

See.
ROS Answers
https://answers.ros.org/question/73627/how-to-increase-rosserial-buffer-size/
My Japanese article
https://qiita.com/PINTO/items/9aa737c284dc4e8212f1#-raspberrypi-と-arduino-へのプログラムの実装

Thanks~ I will try it.


Revit/mep networks

Description & motivation

Changes:

To-do before merge:

Screenshots:

Validation of changes:

Checklist:

[ ] My pull request follows the guidelines in the Contributing guide?
[ ] My pull request does not duplicate any other open Pull Requests for the same update/change?
[ ] My commits are related to the pull request and do not amend unrelated code or documentation.
[ ] My code follows a similar style to existing code.
[ ] I have added appropriate tests.
[ ] I have updated or added relevant documentation.

References

Hey @golembiovskyi , could you pr into this branch instead of release, so I can do some testing locally?

Duplicate of https://github.com/specklesystems/speckle-sharp/pull/1760 (it was created at request of Claire).
Closing this to prevent further git mishaps :)


wrong timing on short texts

VOICE="de-kerstin-low"
NAME=$(mktemp)
echo "Hallo Carola" | /usr/local/share/piper/piper -m /usr/local/share/piper-voices/voice-$VOICE/$VOICE.onnx -f $NAME 2>/dev/null
play $NAME tempo $TEMPO 2>/dev/null
gives out brabbled words. I add the produced wave file below.
Even if the text is extended to "Hallo Carola. Ich habe noch etwas mit Dir vor!"  the "Hallo Carola" sounds strange at best.
piper-example.zip

Same with 1.2.0


move dora to its own top-level package

or perhaps, the disassembly package?

Lancelot/analysis/linear_disassembler & Lancelot/analysis/emu_disassembler?

this functionality will be merged into the various analysis modules in the analysis package


Extended rule-demo with versioned rule files and nicer editable table

Findings from sprint review

And another tip, make sure that you always have the latest master before starting make you changes:
git checkout master
git fetch
git pull


feature request - bundle as a .pkg

i noticed the recent work on the bundling at https://github.com/alexflint/gallium/blob/master/cmd/gallium-bundle/bundle.go
looking awesome !
At the moment the bundling produces a ".app", however its possible with golang to produce a ".pkg" so that the end user gets a decent install experience, and we are developers get control about where things are places and file ACLS assigned etc.
It will also help with getting projects into a CI and CB pipeline, and prepare for continuous updates if we want to integrate that later.
This is the way the golang team does it and its quite simple.
https://github.com/golang/build/blob/master/cmd/release/releaselet.go#L149
It will only build on OSX.
it calls out to pkgbuild & then productbuild for final signing for users.
Its described well here:
http://thegreyblog.blogspot.de/2014/06/os-x-creating-packages-from-command_2.html
I think its very complete.
Please let me know what you think....

Great resources, thanks @joeblew99. We should definitely provide a good bundling option. It seems like there are three possible build artifacts that we might want to output: .app, .pkg, and .dmg. For UI applications that ultimately end up in /Applications, it seems like it's standard to have a .dmg containing a .app. Given that, do you think we also need a .pkg option?

Also, pkgbuild will build a .pkg, but do you know whether that pkg will eventually install the executable go binary directly on the user's computer, or will it ultimately install a .app containing the go binary?


create edit and delete not showing up at top of list

I have been hacking away at Drawer to see if I can integrate it into AOR and it works ok but the create, edit and delete options are not showing up.
I have been trying to figure out how they are instantiated and it is very subtle and hard to track.
Could someone point me in the right direction about where create is declared (I already have the react components that work fine in the non drawer version.)
The nice thing about drawer and   AppNavDrawer is that they are great at controlling the left side precisely. However somewhere in the process the other components are being dropped.

Are you looking for the code in <CrudRoute>?
https://github.com/marmelab/admin-on-rest/blob/6e82e3a6f47b6a74bf513370de30cb67f751e78a/src/CrudRoute.js#L13-L34

I just stepped through all the crudroute code and it is fine. (all the
booleans hasedit hascreate hasdelete are correct)
Therefore it is likely to be failing because of a filters  warning issue
that happens when the component mounts somewhere in the depths of this
function:
markup = ReactReconciler.mountComponent(child, transaction, hostParent,
hostContainerInfo, this._processChildContext(context), debugID);
The warning that is returned is the following and my guess is that the
other components (edit, create etc are ignored)
Warning: Failed prop type: Invalid prop filters of type object supplied
to List, expected a single ReactElement.
in List_ (created by Connect(List_))
in Connect(List_) (created by EqupList)
in EqupList (created by RouterContext)
in div (created by pure(Component))
in pure(Component) (created by RouterContext)
in div (created by BeforeAfterWrapper)
in BeforeAfterWrapper (created by ClearFix)
in ClearFix (created by FullWidthSection_)
in FullWidthSection_ (created by WithWidth)
in EventListener (created by WithWidth)
in WithWidth (created by Master_)
in div (created by Master_)
in Master_ (created by WithWidth)
in EventListener (created by WithWidth)
in WithWidth (created by RouterContext)
in RouterContext (created by Router)
in Router (created by Unknown)
The only difference in code is that I am going through appnavbar instead of
admin/layout.  The filters are the same.
On Wed, Feb 22, 2017 at 1:06 AM, Francois Zaninotto <
<EMAIL_ADDRESS>wrote:

Are you looking for the code in ?
https://github.com/marmelab/admin-on-rest/blob/
6e82e3a6f47b6a74bf513370de30cb67f751e78a/src/CrudRoute.js#L13-L34
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/marmelab/admin-on-rest/issues/344#issuecomment-281610250,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AI7J7XFzBkoBkFb2NbUiS-1UMEEeB5wkks5re_p5gaJpZM4MINdK
.

I figured out the problem.
This is one that is not likely to be repeated very often but it is
something worth thinking about.
As I was combining/ modifying some of the AOR modules there are several
repeated and nonexplicit terms that are used.  For example 'DefaultActions'
is a term used for Actions in  lists, cards and elsewhere.  Some places you
utilize extended names for actions but other places terms like
'defaultState' and 'initialState' are used for the same things.
I spent nearly a week tracking this bug but the bigger issue is the fact
that the call stack for react almost never includes the calling function in
one's own code.  Without specific  and unique names for each variable the
challenge of sorting out the call stack is a  bit of a pain.
If you are interested in the code that uses drawer let me know and I will
publish it.
On Wed, Feb 22, 2017 at 1:06 AM, Francois Zaninotto <
<EMAIL_ADDRESS>wrote:

Are you looking for the code in ?
https://github.com/marmelab/admin-on-rest/blob/
6e82e3a6f47b6a74bf513370de30cb67f751e78a/src/CrudRoute.js#L13-L34
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/marmelab/admin-on-rest/issues/344#issuecomment-281610250,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AI7J7XFzBkoBkFb2NbUiS-1UMEEeB5wkks5re_p5gaJpZM4MINdK
.


Argument "resnet_model" missing in test.py

Hi and thanks for your well-written blog post and code!
I think I encountered a bug in the script for testing. Seems like the argument "resnet_model" is not available in test.py. I'm currently on TF 1.5 so it might be related to this, but as far as I can tell from a quick glance through the code I think it's just a missing argument.
Calling the script as described in the README using:

python test.py --model_id=16645

yields an error:

resnet = getattr(resnet_v2, args.resnet_model)
TypeError: getattr(): attribute name must be string

I was able to fix it by replacing all occurrences of args.resnet_model with "resnet_v2_50" (which is a horrible hack tbh), so I assume the argument is actually just missing in test.py atm.
Cheers
Lukas

Oh, also the used GPU is hard-coded as os.environ["CUDA_VISIBLE_DEVICES"]="3" in test.py. Is there a special reason for this? Works fine for other values. Maybe args.gpu_id could be added along with args.resnet_model.

Thanks for your comments and for pointing out the bug.
It turns out that "resnet_model": "resnet_v2_50" was missing from the data.json file. I uploaded the new data.json.
Regarding the os.environ["CUDA_VISIBLE_DEVICES"]="3" I will remove it from the test.py file. It really does not make sense to be there.

Wow, that was an impressively quick response and fix, thank you!
As a side-comment, calling the test script as CUDA_VISIBLE_DEVICES=X python test.py --model_id=16645 with X being the GPU ID of choice might still be desirable in certain setups (e.g. assigning a specific GPU for testing while others train). So it might still be worth considering adding this argument.


User status

Will have to merge later, I need to implement the client-side, and I am waiting for the websocket initilization to be merged.

Now waiting to merge #57


Install Hive ODBC driver on Azure Function App in App service Plan

Hi,
I want to install ODBC Hive Driver on Azure Function app Deployed in App service Plan.

@shakirshakeelzargar Thanks for reaching out. GitHub forum is primarily for document related issue. Could you help us in posting the link of document that you followed for this issue?

@AnuragSharma-MSFT I did not find any document for How to install Hive ODBC Drivers on Function App. I just came across two Stackoverflow posts stating, this is not possible. For installing Custom Drivers, we need to use VM's.
https://stackoverflow.com/questions/47155223/install-odbc-driver-to-azure-app-service
https://stackoverflow.com/questions/37441052/odbc-or-oledb-database-drivers-under-azure-app-service

@shakirshakeelzargar Thanks for the response. As this is basically a technical query, I would recommend you to post his issue on Microsoft Q&A or StackOverflow for better visibility and it would reach to people who can really provide their valuable inputs that could help you.
Doing this on the forums allows us to help out other people who may have the same questions as you.


[Storage][DataMovement] Update checkpointer to read/write to job file - Part 2

This change is the second part to transitioning the checkpoint read operations to the job plan file instead of the job part plan files. This change handles swapping from IsFinalPart being stored in each job part plan file to storing enumerationComplete in the job plan file. Specifically, this change handles updating the job plan file when enumeration is complete, checking the job plan file when resuming a job to see if we need to re-enumerate, and removing IsFinalPart from job parts all together as it's no longer needed.
The final part is to remove IsFinalPart from the job part plan schema but that will come as part of the larger schema update.

API change check
API changes are not detected in this pull request.


After buffer overflow, input-forward behaves differently

Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.
Describe the bug

After buffer overflow, the forward behaves differently on different nodes.I used fluent-logger, a python tool, to test it. Some of the nodes returned time-out results, while others sent them as normal.
To Reproduce

Close the input-forward at the aggregation end until the agent buffer overflow,I used the following code to test the forward 24224 port.
Some of these nodes return ['Store the Data', {'validation': 'U9fkegEGRJ'}, 'Succeed']
But some other nodes return ['Store the Data', {'validation': 'U9fkegEGRJ'}, 'time-out']
There are also some nodes that return ['Store the Data', {'validation': 'EZr2xcjg8e'}, '[Errno 111] Connection refused']
import random
import string
import time
from fluent import sender

class FluentdTest:
    def __init__(self, fdhost, fdport):
        self.fdhost = fdhost
        self.fdport = int(fdport)
        random.seed(time.time())
        self.ran_str = ''.join(random.sample(string.ascii_letters + string.digits, 10))

    def auto_verify_fluent(self):
        logger = sender.FluentSender('fluentd_test', host=self.fdhost, port=self.fdport)
        cur_time = int(time.time())
        log1 = "Store the Data"
        log2 = {'validation': self.ran_str}
        if not logger.emit_with_time('log', cur_time, {'validation': self.ran_str}):
            log3 =str(logger.last_error)
            logger.clear_last_error()
        else:
            log3 = 'Succeed'
        path = "validation.csv"
        print([log1, log2, log3])
        logger.close()

    def test(self):
        self.auto_verify_fluent()
        # time.sleep(60 * 1.5)
        # self.es_test()
if __name__ == '__main__':
    fdhost = '<IP_ADDRESS>'
    fdport = 24224
    fluentd_test = FluentdTest(fdhost, fdport)
    fluentd_test.test()

Expected behavior

Now I don't know what will happen to forward after buffer overflow. Which of the above situations is normal
Your Environment

Fluentd or td-agent version: 1.9.2
Operating system: Ubuntu 18.04.3 LTS
Kernel version: 4.15.0-72-generic

If you hit the problem with older fluentd version, try latest version first.
Your Configuration
    <source>
      @type forward
      @label @forward_input
      bind localhost
      port 24224
    </source>
    <match **>
      @id forward
      @type forward
      expire_dns_cache 60
      ignore_network_errors_at_startup true
      slow_flush_log_threshold 120.0
      heartbeat_type none
      compress gzip
      <server>
        host fluentd-kafka
        port 24224
      </server>
      <buffer>
        @type file_single
        path /fluentd/log
        total_limit_size 100M
        chunk_limit_size 10M
        retry_type periodic
        retry_wait 60
        flush_mode interval
        flush_interval 15s
        flush_thread_count 1
        retry_forever true
        overflow_action block
      </buffer>
    </match>
    </label>

<!-- Write your configuration here -->

Your Error Log
2020-05-15 09:10:07 +0000 [trace]: [forward] enqueueing all chunks in buffer instance=70033050223740
2020-05-15 09:10:08 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:08 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:08 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:08 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:09 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:09 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:09 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:09 +0000 [trace]: [forward] enqueueing all chunks in buffer instance=70033050223740
2020-05-15 09:10:09 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:10 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:10 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:10 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:10 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:10 +0000 [trace]: [forward] enqueueing all chunks in buffer instance=70033050223740
2020-05-15 09:10:11 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:11 +0000 [trace]: [forward] sleeping until buffer can store more data
2020-05-15 09:10:11 +0000 [trace]: [forward] sleeping until buffer can store more data

<!-- Write your **ALL** error log here -->

Additional context

Should input be suppressed when buffer overflow occurs？
Then the forward should close the socket connection after the buffer overflow，But that's not the case at the moment. The buffer overflowed for a long time and the connection forward reported a timeout error.How should I configure to meet my requirements

Then the forward should close the socket connection after the buffer overflow

If you use default overflow_action throw_exception, in_forward closes the connection.
With overflow_action block, input plugin thread is blocked until resolved buffer overflow. In this case, input plugin thread can't handle incoming connections.


micro_speech example breaks when feeding it trained model on Arduino

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
TensorFlow version (use command below):
1.15
Microcontroller used:
Arduino Nano 33 BLE Sense (Arduino 1.8.12 IDE)

Describe the current behavior
micro_speech example works on pretrained model, but whenever I try to train a model and run it on the device it breaks (even with "yes" and "no" words). The serial port receives the following message:
Feature generation failed
Requested feature_data_ size 536907080 doesn't match 1960.
When running test_micro_speech_test with the same model it also crashes the program (it never stops running).
The model trains fine and achieves a good accuracy on colab and it doesn't give any errors when converting either.
Describe the expected behavior
It should run inference just like the pretrained model.
Standalone code to reproduce the issue
Trained with the following notebook (didn't alter anything):
https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/examples/micro_speech/train/train_micro_speech_model.ipynb
Using the Arduino TFLite library version 1.15

It seems the newest version of the micro_speech training script creates a model that's incompatible with the TF Lite library downloadable in the Arduino IDE. Issue was fixed by building a .zip library with the latest version of TF Lite instead.

Hi do you mind telling me how you built the .zip library? I am a little new to Arduino and cloning the tensorflow source code into a zip file is not working for me.

Hi. You have to build it with the following script (inside the TF GitHub
repo):
tensorflow/lite/micro/tools/ci_build/test_arduino.sh
It takes a few minutes but it will create the zip file at:
tensorflow/lite/micro/tools/make/gen/arduino_x86_64/prj/micro_speech/tensorflow_lite.zip
Hope that helps!
On Tue, 2 Jun 2020 at 23:58, Jared Bell<EMAIL_ADDRESS>wrote:

Hi do you mind telling me how you built the .zip library? I am a little
new to Arduino and cloning the tensorflow source code into a zip file is
not working for me.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/39938#issuecomment-637828042,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIINATIK7Z74D3H26AG6N5DRUVYXDANCNFSM4NM4HK5A
.

What's the simplest way to run shell commands on windows? I tried running it from powershell using sh test_arduino.sh but get a make error. I also tried to run the script on WSL but get a set illegal option error.

I just use WSL. The Linux line endings and the windows ones are different
so it's causing you problems. This is the fix I used.
https://askubuntu.com/questions/966488/how-do-i-fix-r-command-not-found-errors-running-bash-scripts-in-wsl
Just apply dos2unix to the entire micro/tools folder and it will change the
line endings.
On Wed, 3 Jun 2020 at 22:05, Jared Bell<EMAIL_ADDRESS>wrote:

What's the simplest way to run shell commands on windows? I tried running
it from powershell using sh test_arduino.sh but get a make error. I also
tried to run the script on WSL but get a set illegal option error.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/39938#issuecomment-638432690,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIINATLGJPHYMSCGSLLH62LRU2UIDANCNFSM4NM4HK5A
.

@leocorne Hello. I ran test_arduino.sh and got tensorflow_lite.zip.
In example compling is success. but uploading failed.
What progammer did you use in Arduino IDE?

Hi leocorne, it looks like I still have one error getting in the way. When I am generating my zip I get a "No rule to make target 'tensorflow/lite/micro.../fixedpoint.h' needed by generate_hello_world_test_make_project'. Stop. " Any advice on what could be causing this?

@leocorne  Hi, I upload the example in zip library. But it does't work.
I only delete two sparkfun_edge_* files. And I didn't train. Only  used example

I'm also still getting this error, using the latest Arduino Web Editor, but with my own model.  The model was created yesterday using TensorFlow 2.2.0 and the TFLite conversion done online using Google Colab.
I first received this error after using 'micro_speech' from the latest TensorFlow Arduino Library Zip (2.1.0 alpha), via a new clone of Tensorflow.
I appreciate that Arduino Web Editor is using TensorFlow Lite 1.15.0, however as I'm also getting this error using the latest source from the TensorFlow master branch, I don't think its related to the compile toolchain.
I performed the following steps on Arduino Web Editor, using my model:

The model variable names were modified to use the original 'g_model' and 'g_model_len' variable names.
Uploaded my model file, and accompanying .h file.
The speech command word list was increased to include the new trained words and the CommandCount increased from 4 to 13
micro_speech.ino file was modified to use the new model, replacing the old model.

Feature generation failed
Requested feature_data_ size 536907080 doesn't match 1960

I have determined that the error message directly above is the typical behaviour exhibited by the loop() method, when the setup() method has failed to correctly run to completion.  In my case, as the pre-allocation of memory is insufficient in size for the Tensor Arena, or the model input data type is not correct.  There may other reasons also why this has failed.  As the serial monitor may not be connected during the setup() method being evaluated, the associated error messages may not be observed by the developer.
I found these faults in my application by converting the Arduino Web Editor sketch to a Platform IO project, then debugging on the device using a Segger JLink.  I am unaware of any other way to catch these errors at this time.

Hello,
I have the same problem and I can't solve it, could any one please explain me how you do it?
Feature generation failed
Requested feature_data_ size 536907080 doesn't match 1960.

@ErickGQ1423 I think you need to increase the size of the tensor arena.

Thank you @victorromeo, did you solve the problem with this?

Yes. Recently, TFLite Arena Sizes have received additional support, by reporting the size of the actual tensor arena used.
// Guess a larger value of tensor arena 'arena_size' than is actually required.
...
// Build an interpreter to run the model with.
 tflite::MicroInterpreter interpreter(model, resolver, tensor_arena, arena_size, error_reporter);
...
 // This obtains the number of bytes that are actually in use
    return interpreter.arena_used_bytes();

Then change your code, to use the reported value + 100, to allow for future changes to the TFLite Framework implementation.

Hello, @victorromeo could you please help me again?
I have been working for a week with the same problem, but I still have the same. I thing that I am probably not using the correct library in Arduino or probably another code on Colab, I have been following step by step that is in the book TinyML and it looks really easy, but I do not know why my code is not running
I hope you can help me.

Why is this thread closed when the there is no resolution?

This thread shouldn't be closed, nobody brings any solution yet

There is no resolution for this issue yet, it shouldnt be closed.

I was able to solve the issue using the resolution at : https://github.com/tensorflow/tensorflow/issues/44868#issuecomment-845251763
I cloned the latest version of tflite-mirco-examples, added it to the libraries in arduino IDE. Then after adding the lines as per the above resolution, I was able to run a new model on arduino. Also if we train the model using tensorflow v1.15, then  also I was able to run the model on arduino (i did not have to add the additional lines in this case.)


Date and Time Picker

🚀 Feature Proposal
Add a date and time picker to the components library.
Motivation
Date and Time pickers are a common method for entering dates and times into a form. Our date and time picker must support different formats of dates/times and must support entering the date/time via keyboard or the picker.
Example
Start/End time of an appointment.

I can take this one.

@martinrradev this has been assigned to you!


feat: add signal exit

Closes #64
After merging can you please release a new version? i need this to fix stuff on ipfs :)

Currently in the hospital, will do it as soon as I can.


Add "year" placeholder text to year boxes in the publication and creation date fields

A user reported confusion because the year fields for dates do not have "year" in them the same way that "month" and "day" are present in the other fields.

add the html attribute placeholder="year"