idf and tfidf models

Hey,
I'm trying to build idfs and tfidfs models but I cannot store them. I followed the examples and then to build the idfs model I did:
idfs_model = text\
    .TermIdfs()\
    .build(docs)\
    .map(text.TermIdfs.format_item)

and when I try to store it in a file by
data.ObjectStore\
   .Get('models:idfs[wikipedia]')\
   .save_many(idfs_model.collect())

I get this error
sift/models/text.py", line 167, in <lambda>
    .map(lambda (term, (df, rank)): (term, df))\
TypeError: 'int' object is not iterable

For building the tfidfs model I did:
vocab = text.TermVocab(max_rank=100000,min_rank=0,lowercase=True,min_df=5)(docs.sample(False, 0.25))
mentions = text.IndexMappedMentions(sentence_window=1,lowercase=True)(sc, docs, vocab)

tfidfs_model = text\
    .EntityMentionTermFrequency(max_ngram=1, normalize = True)\
    .build(mentions,idfs_model)\
    .map(text.EntityMentionTermFrequency.format_item)

when I try to store it by
data.ObjectStore\
   .Get('models:tfidfs[wikipedia]')\
   .save_many(tfidfs_model.collect())

I get this error:
ValueError: cannot mmap an empty file  caused by:
sift/models/text.py", line 186, in <lambda>
    m = mentions\
ValueError: too many values to unpack

How can i resolve these errors? Is there mistakes in my calls or is there something I should change in the lambda errors?
At the beggining I played around with spark configs but I dont think the problem is there since I succeeded to build the ce and enc models.
I'm using python 2.7.
I'd really appreciate any help.
Thank you in advance,

I would like to post how the issue was resolved it might be helpful for others, since the issues are clearly just a result of multiple commits so I'm sure @andychisholm has a better and efficient solution.
idfs must be in the form of tuples and not dictionaries, so:
idfs= TermDocumentFrequencies().build(corpus)\
         .mapValues(lambda df: math.log(N/df))

Now EntityMentionTermFrequency does not receive mentions in the form of tuple of tuple like (target, (span, text)): (target, text) instead the first part of the build method in EntityMentionTermFrequency are changed to:
def build(self, mentions, idfs):
        log.info('edited')
        m = mentions\
            .map(lambda x: (x["_id"], x["text"])).groupByKey()\
            .map(lambda x: (x[0], list(x[1])))\
            .mapValues(lambda v: ngrams(" ".join(v), self.max_ngram))\
            .flatMap(lambda (target, tokens): (((target, t), 1) for t in tokens))\

Hope this is helpful for someone.
Best,

Thank you for posting this. I found this useful but I am getting an error. Can you pls help resolve.
THese two steps work but I get error in saving the model.
idfs= text.TermDocumentFrequencies().build(docs).mapValues(lambda df: math.log(N/df))
-----------------------works---------------------------
tfidfs_model = text
.EntityMentionTermFrequency(max_ngram=1, normalize = True)
.build(mentions,idfs)
.map(text.EntityMentionTermFrequency.format_item)
------------------------works---------------------------
data.ObjectStore
.Get('models:idfs[wikipedia]')
.save_many(idfs_model.collect())
----------------------------------works------------------------
data.ObjectStore
.Get('models:tfidfs[wikipedia]')
.save_many(tfidfs_model.collect())
--------------------fails-----------------------
Error:
File "/usr/local/lib/python2.7/dist-packages/sift/models/text.py", line 187, in
.mapValues(lambda v: ngrams(" ".join(v), self.max_ngram))
TypeError: sequence item 0: expected string, list found
This is the class updated in sift text.py as per the above comment.
class EntityMentionTermFrequency(ModelBuilder, Model):
""" Compute tf-idf weighted token counts over sentence contexts around links in a corpus """
def init(self, max_ngram=1, normalize = True):
self.max_ngram = max_ngram
self.normalize = normalize
def build(self, mentions, idfs):
    m = mentions\
    .**map(lambda x: (x["_id"], x["text"])).groupByKey()\
        .map(lambda x: (x[0], list(x[1])))\
        .mapValues(lambda v: ngrams(" ".join(v), self.max_ngram))\
        .flatMap(lambda (target, tokens): (((target, t), 1) for t in tokens))\**
        .reduceByKey(add)\
        .map(lambda ((target, token), count): (token, (target, count)))\
        .leftOuterJoin(idfs)\
        .filter(lambda (token, ((target, count), idf)): idf != None)\
        .map(lambda (token, ((target, count), idf)): (target, (token, math.sqrt(count)*idf)))\
        .groupByKey()

    return m.mapValues(self.normalize_counts if self.normalize else list)

Kindly pls let know of any help on this

Hi,
Could you share the output format of this:
Mentions.map(lambda x: (x["_id"], x["text"])).groupByKey()
.map(lambda x: (x[0], list(x[1])))
.collect()

@userofgithub1 I encountered the same problem as @arunzz . It seems that v[0] is a list. So could you please show your full code of training tfidf_model, which might help a lot. Thank you~

Sure, full code is below. Could you please take a look at this issue it's regarding the offset calculations https://github.com/wikilinks/nel/issues/21 thanks :)
import math
import numpy
import ujson as json
from bisect import bisect_left, bisect_right
from operator import add, itemgetter
from collections import Counter

from sift.models.links import EntityVocab
from sift.dataset import ModelBuilder, Documents, Model, Mentions, IndexedMentions, Vocab
from sift.util import ngrams, iter_sent_spans, trim_link_subsection, trim_link_protocol

from sift import logging
log = logging.getLogger()

class TermFrequencies(ModelBuilder, Model):
    """ Get term frequencies over a corpus """
    def __init__(self, lowercase, max_ngram):
        self.lowercase = lowercase
        self.max_ngram = max_ngram

    def build(self, docs):
        m = docs.map(lambda d: d['text'])
        if self.lowercase:
            m = m.map(unicode.lower)

        return m\
            .flatMap(lambda text: ngrams(text, self.max_ngram))\
            .map(lambda t: (t, 1))\
            .reduceByKey(add)\
            .filter(lambda (k,v): v > 1)

    @staticmethod
    def format_item(self, (term, count)):
        return {
            '_id': term,
            'count': count,
        }

class EntityMentions(ModelBuilder, Mentions):
    """ Get aggregated sentence context around links in a corpus """
    def __init__(self, sentence_window = 1, lowercase=False, normalize_url=True, strict_sentences=True):
        self.sentence_window = sentence_window
        self.lowercase = lowercase
        self.strict_sentences = strict_sentences
        self.normalize_url = normalize_url

    @staticmethod
    def iter_mentions(doc, window = 1, norm_url=True, strict=True):
        sent_spans = list(iter_sent_spans(doc['text']))
        sent_offsets = [spns.start for spns in sent_spans]

        for link in doc['links']:
            # align the link span over sentence spans in the document
            # mention span may cross sentence bounds if sentence tokenisation is dodgy
            # if so, the entire span between bounding sentences will be used as context
            sent_start_idx = bisect_right(sent_offsets, link['start']) - 1
            sent_end_idx = bisect_left(sent_offsets, link['stop']) - 1

            lhs_offset = window / 2
            rhs_offset = (window - lhs_offset) - 1
            sent_start_idx = max(0, sent_start_idx - lhs_offset)
            sent_end_idx = min(len(sent_spans)-1, sent_end_idx + rhs_offset)
            sent_offset = sent_spans[sent_start_idx].start

            span = (link['start'] - sent_offset, link['stop'] - sent_offset)
            target = link['target']
            if norm_url:
                target = trim_link_subsection(link['target'])
                target = trim_link_protocol(target)

            mention = doc['text'][sent_spans[sent_start_idx].start:sent_spans[sent_end_idx].stop]

            # filter out instances where the mention span is the entire sentence
            if span == (0, len(mention)):
                continue

            if strict:
                # filter out list item sentences
                sm = mention.strip()
                if not sm or sm.startswith('*') or sm[-1] not in '.!?"\'':
                    continue

            yield target, doc['_id'], mention, span

    def build(self, docs):
        m = docs.flatMap(lambda d: self.iter_mentions(d, self.sentence_window, self.normalize_url, self.strict_sentences))
        if self.lowercase:
            m = m.map(lambda (t, src, m, spn): (t, src, m.lower(), spn))
        return m

class IndexMappedMentions(EntityMentions, IndexedMentions):
    """ Entity mention corpus with terms mapped to numeric indexes """
    def build(self, sc, docs, vocab):
        tv = sc.broadcast(dict(vocab.map(lambda r: (r['_id'], r['rank'])).collect()))
        return super(IndexMappedMentions, self)\
            .build(docs)\
            .map(lambda m: self.transform(m, tv))

    @staticmethod
    def transform((target, source, text, span), vocab):
        vocab = vocab.value

        start, stop = span
        pre = list(ngrams(text[:start], 1))
        ins = list(ngrams(text[start:stop], 1))
        post = list(ngrams(text[stop:], 1))
        indexes = [vocab.get(t, len(vocab)-1) for t in (pre+ins+post)]

        return target, source, indexes, (len(pre), len(pre)+len(ins))

class TermDocumentFrequencies(ModelBuilder):
    """ Get document frequencies for terms in a corpus """
    def __init__(self, lowercase=False, max_ngram=1, min_df=2):
        self.lowercase = lowercase
        self.max_ngram = max_ngram
        self.min_df = min_df

    def build(self, docs):
        m = docs.map(lambda d: d['text'])
        if self.lowercase:
            m = m.map(lambda text: text.lower())

        return m\
            .flatMap(lambda text: set(ngrams(text, self.max_ngram)))\
            .map(lambda t: (t, 1))\
            .reduceByKey(add)\
            .filter(lambda (k,v): v > self.min_df)

class TermVocab(TermDocumentFrequencies, Vocab):
    """ Generate unique indexes for termed based on their document frequency ranking. """
    def __init__(self, max_rank, min_rank=100, *args, **kwargs):
        self.max_rank = max_rank
        self.min_rank = min_rank
        super(TermVocab, self).__init__(*args, **kwargs)

    def build(self, docs):
        m = super(TermVocab, self)\
            .build(docs)\
            .map(lambda (t, df): (df, t))\
            .sortByKey(False)\
            .zipWithIndex()\
            .map(lambda ((df, t), idx): (t, (df, idx)))

        if self.min_rank != None:
            m = m.filter(lambda (t, (df, idx)): idx >= self.min_rank)
        if self.max_rank != None:
            m = m.filter(lambda (t, (df, idx)): idx < self.max_rank)
        return m

    @staticmethod
    def format_item((term, (f, idx))):
        return {
            '_id': term,
            'count': f,
            'rank': idx
        }

class TermIdfs(TermDocumentFrequencies, Model):
    """ Compute tf-idf weighted token counts over sentence contexts around links in a corpus """
    def build(self, corpus):
        log.info('Counting documents in corpus...')
        N = float(corpus.count())
        dfs = super(TermIdfs, self).build(corpus)

        log.info('Building idf model: N=%i', N)
        return dfs\
            .map(lambda (term, df): (term, df))\
            .mapValues(lambda df: math.log(N/df))

    @staticmethod
    def format_item((term, idf)):
        return {
            '_id': term,
            'idf': idf,
        }

class EntityMentionTermFrequency(ModelBuilder, Model):
    """ Compute tf-idf weighted token counts over sentence contexts around links in a corpus """
    def __init__(self, max_ngram=1, normalize = True):
        self.max_ngram = max_ngram
        self.normalize = normalize

    def build(self, mentions, idfs):
        log.info('edited')
        m = mentions\
            .map(lambda x: (x["_id"], x["text"])).groupByKey()\
            .map(lambda x: (x[0], list(x[1])))\
            .mapValues(lambda v: ngrams(" ".join(v), self.max_ngram))\
            .flatMap(lambda (target, tokens): (((target, t), 1) for t in tokens))\
            .reduceByKey(add)\
            .map(lambda ((target, token), count): (token, (target, count)))\
            .leftOuterJoin(idfs)\
            .filter(lambda (token, ((target, count), idf)): idf != None)\
            .map(lambda (token, ((target, count), idf)): (target, (token, math.sqrt(count)*idf)))\
            .groupByKey()
        return m.mapValues(self.normalize_counts if self.normalize else list)

    @staticmethod
    def normalize_counts(counts):
        norm = numpy.linalg.norm([v for _, v in counts])
        return [(k, v/norm) for k, v in counts]

    @staticmethod
    def format_item((link, counts)):
        return {
            '_id': link,
            'counts': dict(counts),
        }

Thanks！ But there are still problem. I print out a middle value of m
m = mentions\
    .map(lambda x: (x["_id"], x["text"])).groupByKey()\
    .map(lambda x: (x[0], list(x[1])))
print m.take(1)

which is
[(u'en.wikipedia.org/wiki/Goritsky_Monastery_(Pereslavl-Zalessky)', [[0, 99999, 4871, 3, 457, 99999, 6238, 8, 2, 1131, 4434, 4871, 1, 99999, 1343]])]
I can't figure out what this part mean.
.mapValues(lambda v: ngrams(" ".join(v), self.max_ngram))\

Your output should look like this after print
.map(lambda x: (x[0], list(x[1]))):
[(u'en.wikipedia.org/wiki/Dru_Drury', [((20, 29), u'It was described by Dru Drury in 1770.'), ((20, 29, 2), u'It was described by Dru Drury in 1770.2')]), (u'en.wikipedia.org/wiki/Cerambycidae', [((61, 73), u'"Plinthocoelium virens" is a species of beetle in the family Cerambycidae.')])]
The line your asking about, sends the joined strings to the ngram method.
Hope this is helpful :)

Hi,
I'm trying to follow your steps, but I'm getting an error when initializing  mentions . When I do  mentions = text.IndexMappedMentions(sentence_window=1,lowercase=True)(sc, docs, vocab) I'm getting:
AttributeError                            Traceback (most recent call last)
<ipython-input-13-e8a4ec6995b7> in <module>
----> 1 mentions = text.IndexMappedMentions(sentence_window=1,lowercase=True)(sc, docs, vocab)
      2

/var/lib/spark/tmp/spark-7e3c1a98-7e9a-40b5-9d8f-ff01ef93065e/userFiles-ed742db8-39c1-4a20-9871-8f87c7c11075/sift.zip/sift/dataset.py in __call__(self, *args, **kwargs)
      5
      6     def __call__(self, *args, **kwargs):
----> 7         return self.build(*args, **kwargs).map(self.format_item)
      8
      9     def build(self, *args, **kwargs):

/var/lib/spark/tmp/spark-7e3c1a98-7e9a-40b5-9d8f-ff01ef93065e/userFiles-ed742db8-39c1-4a20-9871-8f87c7c11075/sift.zip/sift/models/text.py in build(self, sc, docs, vocab)
     93         tv = sc.broadcast(dict(vocab.map(lambda r: (r['_id'], r['rank'])).collect()))
     94         return super(IndexMappedMentions, self)\
---> 95             .build(docs)\
     96             .map(lambda m: self.transform(m, tv))
     97

AttributeError: 'NoneType' object has no attribute 'map'

Could anyone help me, please?
Thanks

Hi, I'm dealing with the same error. My output is the same as @woshiyyya , how can I do it to get this output @userofgithub1 ?
Thanks


Added MeteorObservables and MongoObservables wrappers with RxJS

@Urigo
Usage example for Collection and Cursor:
let collection = new MongoObservable.Collection("test");
let ob : Observable = collection.find({});
ob.subscribe((col) => {
   console.log("new data:", col);
});

And then in the View:
<div *ngFor="let item of ob | async"></div>

Usage example for Meteor.call:
MeteorObservable.call("myMethod").subscribe((response) => {
   console.log("server response:", response);
}, (error : Meteor.Error) => {
   console.log("Error!", error);
});

Usage example for Meteor.subscribe:
MeteorObservable.subscribe("myMethod").subscribe(() => {
   console.log("ready!");
}, (error : Meteor.Error) => {
   console.log("Error!", error);
});

@dotansimha  wow! Will that work in a service and reflected in a component?

@Hongbo-Miao @barbatus @kamilkisiela

@Tallyb Yeah, you can also use ngrx to manage your application flow and put the subscriptions in a service instead of the Component.
@Urigo do you think we should add this as default when using MeteorComponent? (the call and subscribe).

@dotansimha how can I import MeteorObservable, MongoObservable? Thanks

@dotansimha after I add
import { MongoObservable } from 'angular2-meteor';
export const Tests = new MongoObservable.Collection('test');

It give me error in my terminal and crash my server:

Error: Cannot find module 'meteor/mongo'

@dotansimha No more Cursor :fire: Nice job!

@Hongbo-Miao
That's weird. which version of Meteor do you use?
Also, can you share the project so I can try it locally?
I took the angular2-meteor-boilerplate and tried with it, and it works great (the imports).
I can also try to remove the imports from the code, but it will mean that we depend on Meteor's globals and I do not want to.

@Hongbo-Miao
Also, take a look in here: https://github.com/dotansimha/angular2-meteor-with-observables
This is a simple example for usage, and it works great...

@Hongbo-Miao @Tallyb @Urigo @kamilkisiela
I need your advice about the usage of call and subscribe.
At the moment, when you use those methods, you get in response the RxJS Observable, which does not run the actual Meteor method with subscribing to the result.
That means, if you use it like this:
MeteorObservable.call("...");
MeteorObservable.subscribe("...");

It would never execute the Meteor method or subscription, instead, you must register to the data as a subscriber:
MeteorObservable.call("...").subscribe((res) => {}, (err) => {});
MeteorObservable.subscribe("...").subscribe((res) => {}, (err) => {});

As a solution, I can subscribe to the Observable before it returns from the method call, and it will  make sure that the method actually called without any subscription.
What do you think?

@Hongbo-Miao The imports issue fixed in #20df9803cac8ee0ad3a4014bd66ed965a5938165.

@dotansimha  assuming you will do the subscribe to the observable yourself, what would the call look like?
Meteor.observable.call ("myMethod", (res)=>{}, (err)=>{})
If so, I am in favor of the original way: .subscribe.

@dotansimha how to unsubscribe a Meteor.SubscriptionHandle now? I didn't get it here.

@dotansimha is there a way to unsubscribe a Meteor.SubscriptionHandle now? I didn't get it here.
If not, maybe add something like MeteorObservable.unsubscribe('products')?

On usage in MeteorComponent. We could think actually here about a different approach we already talked about: splitting MeteorComponent.
One way to implement it could be to have, for example, a CallObservable annotation and ICallObservable interface, so one could use it as:
@CallObservable()
class MyComponent implements ICallObservable {
  click() {
     this.call(...).subscribe(...);
  }
}

@dotansimha linter seems to throw a bunch of errors, do you mind to fix? )

@Hongbo-Miao , the ngrx branch still need some modifications and fixes, because I try to implement it with effects package.
Note that the line you mentioned in the code will work with ngrx because of you use effects it will execute and subscribe to the data, for example:
  @Effect() subscribeProducts$ = this.updates$
    .whenAction(ProductsActions.CALL_SUBSCRIBE_PRODUCTS)
    .switchMap(() => this.productsService.subscribeProducts())
    .map(() => ({type: ProductsActions.LOAD_PRODUCTS_FROM_COLLECTION}));

  @Effect() loadProducts$ = this.updates$
    .whenAction(ProductsActions.LOAD_PRODUCTS_FROM_COLLECTION)
    .switchMap(() => this.productsService.getProducts())
    .map(products => ({type: ProductsActions.PRODUCTS_COLLECTION_UPDATED, payload: products}));

So in that case, the CALL_SUBSCRIBE_PRODUCTS uses the subscription and when it's ready it dispatch the LOAD_PRODUCTS_FROM_COLLECTION which return an Observable with the data, and each time the data changes, the PRODUCTS_COLLECTION_UPDATED called.
If you don't use ngrx, you will need to subscribe to the data by your self (if you use ngFor with async pipe you won't need).
The subscription handler of Meteor subscribe is inside the Observable you get, and when you dispose or unsubscribe, it will automatically close the Meteor subscription.
So if you use ngFor to display the data, and you hide the ngFor using ngIf - the subscription will close it self and reopened when needed.
Also, when the Component destroyed it will close the subscription.
Regarding the Tracker.autorun, can you provide a use case and I will try to find a solution?
Thanks.

@barbatus Fixed!

@Hongbo-Miao , I fixed the ngrx branch and added example for call and example for iterating using ngFor.

@dotansimha I just want to say WOOOW. It saves tons of codes.
This line you didn't handle unsubscribe part in the Effects, can you add that part? Thanks

@dotansimha oh, for Tracker question, I realize right now it is already reactive by default.
However, a lot of time I want it be NOT reactive.
When you dispatch ADD_PRODUCT_SUCCESS, the app gets PRODUCTS_COLLECTION_UPDATED immediately and show on the View.
So now is

ADD_PRODUCT_SUCCESS -> PRODUCTS_COLLECTION_UPDATED

What I want a lot of time is when you dispatch ADD_PRODUCT_SUCCESS, it won't dispatch PRODUCTS_COLLECTION_UPDATED until I dispatch  LOAD_PRODUCTS_FROM_COLLECTION manually.

ADD_PRODUCT_SUCCESS -> LOAD_PRODUCTS_FROM_COLLECTION (by me) -> PRODUCTS_COLLECTION_UPDATED

Regarding the non-reactive collection, I guess that we can change the implementation, It's just the difference between registering to the observe of the Mongo.Cursor.
I have some thoughts about this:
How to trigger the reload action? I mean, all you get from MeteorOvservables is an Observable which you can't manually trigger to reload.
Possible solution is maybe to inherit from RxJS's Observable into ObservableMeteorCollection (this will also simplify the current implementation that declare and bridge each method of the original Mongo.Cursor), and extend it with method like reload and maybe even stop to provide easy use of unsubscribe.
Maybe the creation can be like this (not completely sure about the implementation, please tell me what you think):
// Reactive
let observable : Observable<any> = MyObservableCollection.find({});

// Non-reactive
let observable : Observable<any> = MyObservableCollection.find({}).nonReactive();
// And then, to reload:
observable.reload();
// And, to unsubscribe
observable.stop();

Regarding the Tracker thing, I think about a Meteor subscription that uses ReactiveVar and how to re-create the subscription when needed... It might be a little tricky and we need to think about the implementation and how to keep it simple as possible.

Current progress released as 0.6.3-beta.2

@Hongbo-Miao Regarding the unimplemented unsubscribe thing... not sure how to approach it, because we do not have access to the return value of the subscribe because @ngrx/effects is in charge of the subscribe call, and I'm not sure how we can get it :( any idea?

It makes no sense that it is a minor release. Should be 0.7.0-beta.1 no?
On Sat, Jul 30, 2016, 19:58 Uri Goldshtein<EMAIL_ADDRESS>wrote:

Current progress released as 0.6.3-beta.2
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Urigo/angular2-meteor/pull/358#issuecomment-236376354,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHC0j9KSwlr9Ee2Wat95ZzspSZcXrW1Yks5qa4KygaJpZM4JXGBG
.

@Tallyb makes sense, I'll release the next beta as such

@dotansimha your implement of the non-reactive collection is good to me I think.
For unsubscribe, I am not clear about

we do not have access to the return value of the subscribe because @ngrx/effects is in charge of the subscribe call

If you do not use ngrx/effects, how do you unsubscribe(Meteor Collection) in your the API right now?
(BTW, please write clear unsubscribe (RxJS) or unsubscribe (Meteor Collection), sometimes I don't know which you unsubscribe you are talking about)

Yeah sure, sorry about the confusion.
If you use without effects you need to do:
// This will return Observable
let observable = MeteorObservable.subscribe("mySub");
// This will return the actual Subscription
let subscription = observable.subscribe(...);

// And to destroy it, you can use:
subscription.unsubscribe(); // This will trigger the SubscriptionHandle.stop() of Meteor

But if you use effects, you do not have access to the Subscription object (the one we get from let subscription = ..., so you do not have the variable to call unsubscribe on.
You have an Observable object in the variable you put the @Effect decorator on, but I tried to call unsubscribe on that, and it did not call the unsubscribe of the relevant Subscription that we need.
Any idea?

@dotansimha if Meteor has an API, Meteor.unsubscribe('products'), everything will be easy... You can implement a MeteorObservable.unsubscribe('products').
Let me ask some help from ngrx. Maybe there is way.

@Hongbo-Miao that's not good... As far as I know, Meteor does not have Meteor.unsubscribe, also, you can subscribe to the same publication multiple times with another arguments.
I can keep the Meteor's SubscriptionHandler and expose unsubscribe of our own, but I think that we should stick to find a solution using RxJS or ngrx...
All that we need is to get access to the actual Subscription object that switchMap gets...
I am trying now to wrap RxJS Observable in order to provide nonReactive, stop and reload.

@Hongbo-Miao please refer to my new PR: https://github.com/Urigo/angular2-meteor/pull/362

@dotansimha  @Hongbo-Miao  I am wondering if what you call reactive and non reactive observables, is in fact hot and cold observables, and if the solution described here https://github.com/Reactive-Extensions/RxJS/blob/master/doc/gettingstarted/creating.md can be of help.

@Tallyb , did not think about the hot and cold. it might solve the issue the force you to run "subscribe" for the return value of "call" and "subscribe".
I will take a look at this now.
Thanks!

One of the biggest issues I have with this (and this seems to be a growing problem in general now that Meteor is trying to support multiple view libraries and npm) is that by creating a new type called MongoObservable, you've removed ecosystem support for things like SimpleSchema (object validation is a critical piece to any not-small-todos-like-app).
You'll probably want to re-tailor MongoObservable to behave like a default Mongo.Collection if you want to have any widespread usage to this.  I like the idea, but I hate that in order to use this it breaks a bunch of other stuff.

@denodaeus I don't understand why you have issue with that.
Your collection is still the same, create it:
let observableCollection = new MongoObservable.Collection(...);
let theMongoCollection = observableCollection.getMongoCollection();
// now attach you schema to the theMongoCollection

And then you have a wrapped collection, and your cursors will look the same...

Yep I just realized that by looking at your changes, see edited note on my comment above.
Thanks for the awesome updates, one more step closer to a testable angular2-meteor app :)

Hi!
I have a question concerning performance.
If I subscribe to the MeteorObservable like in your example products.service in the constructor:
  constructor() {
    this.data = ProductsCollection.find({});
    MeteorObservable.subscribe("products").subscribe();
  }

AND this service is injected in the root injector (e.g. because it provides data needed in serveral modules). In this case, isn't the subscribe "hot" all the time and consuming ressources on server and client?
What is your recommended best practice? Always inject "data"services only local to the component which is in need of the data? Or write "initialize" and "stop" methods in which the MeteorObservable subscritions are started and stopped?

@stefanholzapfel
Yeah that's correct.
You can either create a function in the service that subscribes to the Meteor subscription and then call it when you want to do the actual subscription, or you can use ngrx package to create a flow that subscribes only when you need to (take a look at the examples here)

Cool thx.
As far as I now, stopping a meteor subscription would normally delete any coresponding documents in clients minimongo. Do you now if it's possible to leave that data as a "cache" in the minimongo? Maybe even after browser restarts?

Hey guys!
Sorry for bothering you again, this is my next try of an implementation of a global data service:
@Injectable()
export class ItemService {

    private itemParameters = {
        find: {},
        options: {limit: 10}
    };
    private itemObservable: ObservableMeteorSubscription;

    constructor() { }

    public stopSync() : void {
        this.itemObservable.stop();
    }

    public getItems(limit?: number) : Observable<Item[]> {
        if (limit) {
            this.itemParameters.options = {limit: limit};
        }
        const newItemObservable = MeteorObservable.subscribe('item', this.itemParameters);
        newItemObservable.subscribe();
        if (this.itemObservable) {
            this.itemObservable.stop();
        }
        this.itemObservable = newItemObservable;
        return itemCollection.find({});
    }
}

Do you think that is a reasonable approach?
I was particularly wondering, if it makes a difference whether I first subscribe to the publication with the new parameters and afterwards stop the old subscription or if the order doesn't matter at all? My naive assumption was, that if I first subscribe with the new parameters, the "overlapping" records in the minimongo wouldn't be deleted when stopping the old subscription and I wouldn't have to reload them?
Thanks very much!!

@stefanholzapfel In this case the order doesn't matter (actually stopping is even not needed here).  Meteor is smart enough to recognize that same subscription is made with different params to stop the previous one.

You're sure? I have tried out to subscribe with a limit of 10, afterwards (without canceling first subscription) I subscribed with a limit of 5, and server side changes (with mongo cli) on records with ranks higher than 5 in the document were still reflected to minimongo...

Ok, this seems to work only when a subscription is wrapped in the autorun (https://guide.meteor.com/data-loading.html#changing-arguments).
This is what happens when you re-subscribe https://guide.meteor.com/data-loading.html#publication-behavior-with-arguments. I think the order doesn't matter much when to stop the previous.

Thx, my hope was to get along without making the service a MeteorComponent :)

Well, it's MeteorComponent because it's for components, since it implements OnDestroy and run ngZone on callbacks.  In case everything else, I don't see why it shouldn't be done with direct Meteor API, it's just same as MeteorComponent's one. Observables should be helpful when you use ngrx for example, for other case it is not always true.

When I try to apply an options object on the MongoObservable.Collection('item').find() method, I get an error.
MongoObservable.Collection<Item>('item').find({}, {})
works, but returns ALL elements in the local minimongo-db which is not always what I want
e.g.
MongoObservable.Collection<Item>('item').find({}, { limit: 10 })
throws me:
"Error: must use ordered observe (ie, 'addedBefore' instead of 'added') with skip or limit"

I posted one issue last month nobody replied to that. Can any of you please check if i doing anything wrong: https://github.com/dotansimha/angular2-meteor-with-observables/issues/9


[test-case-reporting] Add endpoint for cron task

Closes https://github.com/ampproject/amp-github-apps/issues/892

Pushing to unblock.
@estherkim we can address any comments in a future PR


flag to disable implicit dynamic in generic type annotations

no-implicit-dynamic still allows things like:
class C<T> { T t; C(this.t); }
void main() {
  C c = new C(42);
  c.t.oops();
}

If we require that to be written as C<dynamic> it makes the dynamic dispatch a bit more obvious. I think what we'd want to do here is interpret C as C<Object>, that way the simple syntax can still be used, but it becomes safer statically if someone is using this flag.
Context https://github.com/dart-lang/sdk/issues/25573#issuecomment-228823689

CC @leafpetersen @munificent  -- I left this one lower priority as I'm not sure if we have a current need for it.

Agreed, probably lower priority in my mind.

@zoechi just hit this as well, so I bumped the priority

no-implicit-dynamic now produces an error on the code in the description:
$ cat 26784.dart
class C<T> { T t; C(this.t); }
void main() {
  C c = new C(42);
  c.t.oops();
}
$ dartanalyzer --no-implicit-dynamic 26784.dart
Analyzing 26784.dart...
  error • Missing type arguments for generic type 'C<dynamic>' at 26784.dart:3:13 • strong_mode_implicit_dynamic_type
1 error found.
$ dartanalyzer --version
dartanalyzer version 2.0.0-dev.63.0


Add logging level for RNPM previous linking

Sometimes when working with a team of developers, someone adds a library but does not link it.  To identify if this required linking, you have to type react-native link and then read a wall of text to see if rnpm handled the issue.   This can be sped up with a friendly logging level for previously linked messages.
This PR turns this:

Into this:

Which is a pretty nice experience!   This also helps you quickly identify when 1 thing wasn't linked, at a glance, like so!

By analyzing the blame information on this pull request, we identified @grabbou to be a potential reviewer.

Looks like a test failed, but unrelated to the change as far as I can tell.  Seems to be a timeout.

I think info, debug, warn and error are the only log levels used everywhere. So we shouldn't change that. I'd prefer something like,
install info [skip] ...
install info [done] ...

But @Kureev is the boss!

The color adjustments as you see are quite stark, and helpful.  Moving to a prefix won't have those benefits (unless the prefix is colored).  Happy to adjust this either way, though :)

I think we can color the prefix. But let's wait for @Kureev

Great PR, I think skip prefix of gray color would be good enough (and also potentially useful for other commands).

@grabbou - Here's what chalk.grey looks like on my machine.   Seems pretty good.

Let me know if I should update the PR with this.

Looks good to me that way. Don't want to spend much time working on the log module as eventually we are going to replace it with something unified across all commands :)

ok @grabbou - updated the PR to be the grey.

@GantMan updated the pull request - view changes

@GantMan updated the pull request - view changes

@GantMan updated the pull request - view changes

@GantMan mind updating just to be sure it passes? I'll ship it right after! Thanks, looks very good now :)

@GantMan updated the pull request - view changes

@grabbou  - tests failed again?   Any way you can just kick it off from circle-ci?

You can ignore the tests, I think they are failing because of other reason. Have you tried rebasing?Github says there's a conflict (I just moved files around in my recent commit).

@GantMan updated the pull request - view changes

@GantMan updated the pull request - view changes

@grabbou - updated the PR to be OK with master.

@facebook-github-bot shipit

@GantMan this PR has jest test failures.
@grabbou, please keep an eye on failed CI reports when shipping.
I'll do a quick revert, please send give it another try.
Unless you have a quick fix in mind

Filed PR builds https://circleci.com/gh/facebook/react-native/tree/pull%2F9507

@bestander  - as you can see from the above, we're concluding that the failures are not directly related to this PR.  If you have a suggestion for how to fix this, happy to do so.

Yeah, we do have random errors or integration between build systems broken, this week was particularly red build-wise because of Buck changes.
But when both circle and travis are red then it is a good chance that something is wrong with the code.
In this case, build before revert
https://circleci.com/gh/facebook/react-native/10441 (failing on npm test)
after revert
https://circleci.com/gh/facebook/react-native/10442 (green so far).
Could you help us out and give find a fix?
I suppose that  local-cli/link/link.js is used in jest tests and loading chalk just chokes jest.
Most logical is to mock chalk in mocks folder

Yeah, we do have random errors or integration between build systems broken, this week was particularly red build-wise because of Buck changes.

I think that's partially the reason we tend to ignore errored builds when changes are small. Sorry for the troubles, I'll double check the trace next time!
FYI I'll be online in 2/3 hours to take a closer look at that one.

Ah, that makes sense @bestander
What's the procedure from here?  Do I open a new PR with the mocks?

Yeah, just open a new PR please


[Bug] Overwriting Azure::Core::Http::Details::c_GetStreamForBody to false in Context for chunked response does not work.

When I call:
context = context.WithValue(Azure::Core::Http::Details::c_GetStreamForBody, false);

before a request is sent that's returning a chunked response with the context, the response will contain no response body, resulting in RawResponse::GetBody() returning empty buffer.

this is because we didn't check the bool value here. https://github.com/Azure/azure-sdk-for-cpp/blob/ef0c530ee0250ed879cb68c37ca11c2b2ad7e249/sdk/core/azure-core/src/http/transport_policy.cpp#L19

No need to be fixed.
Context is not used any longer to set DownloadViaStream.
Instead, HTTP Request is used to set this setting.


Why jnsaf is not working as it said in JN-SAF article?

I have tried multiple times to test jnsaf against malwares like triada or UpdtKiller but i didnt get any results back. it does not even passed to nativedroid for analysing *.so files.
i dont know whats the problem! can you tell me the exact malware hash's that had been tested with jnsaf?
thank you

Problem Solved. I just increased the timeout threshold and waited for JN-SAF to finish analyzing the java part.


⚠ Bofrost.lu has degraded performance

In 03c2f82, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 888 ms

Resolved: Bofrost.lu performance has improved in 72a837e after 8 minutes.


add cookiecutter-template to Python templates

Current coverage is 99.65%

Merging #664 into master will not affect coverage as of 38d3d61

@@            master    #664   diff @@
======================================
  Files           13      13
  Stmts          577     577
  Branches         0       0
  Methods          0       0
======================================
  Hit            575     575
  Partial          0       0
  Missed           2       2

Review entire Coverage Diff as of 38d3d61

Uncovered Suggestions

+0.35% via ...ookiecutter/utils.py#27...28

Powered by Codecov. Updated on successful CI builds.


Are the audio books supposed to work? And also in the features it says that "Hiding podcasts, episodes and audiobooks from the homepage" is optional so how do I turn this off to get access to the audiobooks?

🧐 Have you read the FAQ and viewed similar issues?

[X] I have read the FAQ
[X] I have checked older issues, open and closed
[X] I did a clean install of SpotX as described in the FAQ

Сountry of your account
united states
What is your Spotify Plan?
Free
Do you have Spicetify installed?
Yes
ℹ Computer information
- Spotify: <IP_ADDRESS>9.gf1bb1e36 (couldn't find)
- Windows: Windows 11 21H2 22000.527 (don't have pause/break button - couldn't find)
- PowerShell: 5.1.22000.832 (did get this one though :)

📝 Description

click on the search bar (worked fine - what I expected)
click on audiobooks (worked fine - what I expected)
choose an audiobook and click on it (doesn't work - not what I expected)
it loads and then says "couldn't find that page"
...

📸 Screenshots

Fixed
update SpotX


add cache control header to tlon-hosted uploads

I am investigating how we can add some caching to tlon-hosted assets. The easiest/most straightforward way is to add the cache-control header to the asset while uploading it. This needs to be done when creating the upload URL and when uploading the asset from the client, which is what this PR attempts to do.
I found the places in this PR that reference upload headers, but I need another set of eyes to guide me here — in the approach as well as development process.
PR Checklist

[ ] Includes changes to desk files
[ ] Describes how you tested the PR locally (test ship vs livenet)
[ ] If a new feature, includes automated tests
[ ] Comments added anywhere logic may be confusing without context

That's great! Thanks for taking a look!

@latter-bolden I'm marking this as do-not-merge for now. I need to think a little more about this — deploying memex with the cache changes is instant while rolling out app changes is gradual, and this will break uploads for any clients without the corresponding cache headers.

Confirmed after the latest hosting-side changes that this continues to work as expected in the staging environment

Going to wait to merge this until after we cut for the next native app tester release.


Beam search v2

Description of changes:
Related Issues:
Linked Issues:

Issue #

Issues closed by this PR:

Closes #

Before merging:

[ ] Did you update the flexflow-third-party repo, if modifying any of the Cmake files, the build configs, or the submodules?

fix topk. add beam slot


Use src layout

Closes #137 -> avoid including unneeded packages in distribution

@krahabb  if you are willing and able to test this PR with your workflow to see if #137 is resolved, it would be very useful.


Add swig bindings for field element mapping to G1 and G2

In trying to use jblst (and therefore blst) for EIP-2537 implementation in Hyperledger Besu's besu-native project, I found that blst_map_to_g1 and blst_map_to_g2 were not exposed via SWIG.
I suspect there was a reason that these functions were left out of the swig bindings, but I don't know why.  To get a complete implementation, I added them (disclaimer: not a C programmer nor familiar with SWIG).
Using BLST java bindings is the fastest implementation we have tested, so we are motivated to use it as the basis for EIp-2537 in besu.  I did verify that java python and rust bindings still build correctly, but only tested Java bindings.
If this should go in blst_aux.h or elsewhere, or if there is another way to map field elements without adding these methods to the interface, LMK.  TIA

I suspect there was a reason that these functions were left out of the swig bindings, but I don't know why.

Field types are not exposed though the SWIG interface, only EC points and fp12. Or in more practical terms the application code has no way of constructing arguments for the blst_map_to_g{1,2}. Which is why they are masked with #ifndef SWIG. But note that the #ifndef in question controls whether or not masked interfaces will be available on the application side of the SWIG interface, but not whether or not one can invoke them from the glue C++ code. In other words you didn't need to remove the #ifndef SWIG in blst.h in order to call them from blst.hpp.

EIP-2537

Then wouldn't it be appropriate to accommodate it with a method that consumes and produces data in the corresponding format? The thing about most of the bindings is that less transitions between languages are better. So why not something like static void eip2537_map_to(byte out[128], const byte u[64]), which in Java (and others) would look as x = P1.eip2537_map_to(u);. Well, this asks for a question why not accommodate remaining primitives, which would be a somewhat larger project than a normal support issue... Maybe we can compromise and limit the scope to adding EIP-2537-specific serialization and deserialization... To summarize, I'd suggest to close this request and open an issue instead to hash out the goal first, before moving on to implementing things.

Thx for the explanation.  Closing this, and weighing the options of opening an issue asking for 2537 specific serialization/deserialization versus just implementing a c-wrapper in besu-native to call blst.
Other operations like MultExpr currently require considerable back and forth that would be nice to limit as well.

Have a look at #223 as an initial step.


Implementation with react JS

Any Documentation how to attain this with react JS an react SSR?

All our apis are HTTP based, and you would need key id and secret to use the API. Its not recommended to use them directly in browser.


update-debug-edid help and error

I think #418 can be resolved by the following commit.

#418 isn't fully resolved, but this is a step in the right direction.   It does make things better.
I think.
I'll have to admit I'm having trouble envisioning the details, but here are my thoughts:
I don't think the new error message is better.  It still seems misleading.
H2U 00:17:45>debug edid o0
no such port
H2U 00:17:45>debug edid input0
no such port
H2U 00:17:45>debug edid foo
no such port

the first one should not error :p
input0 and foo should have different error messages.  maybe.
"invalid parameter" might be better.  It doesn't tell me anything new, but maybe it doesn't mislead me.
I am also not sure how much value there is for this.

@CarlFK  As far as I can understand, error log can be divided into the following sub-cases:
H2U 00:17:45>debug edid o0
o0 has no EDID capabilities.
H2U 00:17:45>debug edid input0
input0 is not an output port.
H2U 00:17:45>debug edid foo
invalid parameter
H2U 00:17:45>debug edid  output1
no error message(in case output1 has EDID capabilities.)
Am I correct?

kinda.
Not sure that calling them categories helps.
This is a bug:
H2U 00:17:45>debug edid o0
o0 has no EDID capabilities.

These two are similar:
input0 is not an output port.
H2U 00:17:45>debug edid foo
invalid parameter

and "no error" is also different.
otherwise... sure. ;)

@shivmgg Hi! Could you make the changes requested by @CarlFK and then I can merge your pull request!

Hi, I got a little busy with OpenSoCDebug Project. I will update it by tomorrow.


Community partner project count inconsistent

When I view a partner on the community partner map and look at the projects count (right clicking) the number is different than what shows on the total number of projects in the community partner project report. For example, City of Omaha.

@MansiKamthan @SachinPawaskarUNO this is still not working at least from the CAT view. Projects are inconsistent between map and community partner report.

The map data is updated from production so please compare the count with production report. it should match unless the project was added to the community partner in past 3 weeks.

Double checked with @MansiKamthan on 3-25-2024 that this is working in production and matches across the map and community partner report.


chore(main): release 1.6.7

:robot: I have created a release beep boop
1.6.7 (2023-10-17)
Performance Improvements

📝 去除冗余的依赖项 (a6744dc)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/viarotel-org/escrcpy/releases/tag/v1.6.7 :sunflower:


Fix yaml indent in key value case

Closes #8740
This PR resolves unexpected indent in a new line.
I rollback cause fix in https://github.com/vim/vim/commit/acc224064033e5cea21ef7f1eefb356ca06ff11d
However, I don't understand the purpose of this fix.
Please check that this rollback has no problem.
Current behavior (| is cursor)
a: b
  |

Updated behavior
a: b
|

This problem doesn't occur in Vim 8.1. (8.1.3741)
I confirmed diff between codes in 8.1 and 8.2.
The following fixes affect this problem.
https://github.com/vim/vim/commit/acc224064033e5cea21ef7f1eefb356ca06ff11d#diff-723cc09b2d0f8f9383c670a8846979363fbc67a741aad1aae657242b8c2667a3L56
acc2240#diff-723cc09b2d0f8f9383c670a8846979363fbc67a741aad1aae657242b8c2667a3L65
In 8.1, process doesn't enter any if/elseif prevline block in GetYAMLIndent function.
In 8.2, process enters the last elseif prevline block in GetYAMLIndent function. (https://github.com/vim/vim/blob/master/runtime/indent/yaml.vim#L145)

The problem is resolved by rollbacking either one line.
Just in case, I rollbacked both lines.

AFAICS the change was introduced by this PR.

I guess this test is wrong.
https://github.com/vim/vim/blob/b4011afe53ce2d065e49718b8c5b9974b7022f92/runtime/indent/testdir/yaml.in#L16-L19
Multiple line value should be set | or > at the top.
I fixed this test.


fix: button theme types

:tada: This PR is included in version 5.19.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


p5 Renderer

I've noticed that Matter.js uses a context-based renderer, but I'm more used to p5 rendering techniques than vanilla context. Could you add an optional p5 rendering module? Basically a replica of the render module, but p5 friendly.

If you're interested in using the engine with p5 there are some third-party integrations, examples and tutorials worth checking out:
https://github.com/CodingTrain/p5-matter
https://github.com/b-g/p5-matter-examples
https://github.com/pzp1997/p5-matter
https://github.com/liabru/matter-js/wiki/Tutorials
Integration is out of scope for this project though, but hope that helps you get started.

I recommend adding p5 rendering support where instead of a 2dcontext, it's
a Canvas or Graphics Buffer.q
On Sun, Feb 7, 2021, 12:42 PM liabru<EMAIL_ADDRESS>wrote:

Closed #968 https://github.com/liabru/matter-js/issues/968.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/liabru/matter-js/issues/968#event-4300473222, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AO4EAS2EYKRCMCJR2PGN2YDS53GH7ANCNFSM4XGVDVCQ
.


SyntaxError: invalid syntax (jupyter, anaconda, windows10)

`import matplotlib
matplotlib.use("Agg")
import matplotlib.pylab as plt
%matplotlib inline
import IPython.display as ipd
import sys
sys.path.append('waveglow/')
import numpy as np
import torch
from hparams import create_hparams
from model import Tacotron2
from layers import TacotronSTFT, STFT
from audio_processing import griffin_lim
from train import load_model
from text import text_to_sequence
from denoiser import Denoiser Traceback (most recent call last):
File "G:\anaconda\lib\site-packages\IPython\core\interactiveshell.py", line 3291, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 13, in
from model import Tacotron2
File "G:\waveglow\tacotron2\model.py", line 6, in
from utils import to_gpu, get_mask_from_lengths
File "G:\waveglow\tacotron2\utils.py", line 31
x = x.contiguous().cuda(async=True)
^
SyntaxError: invalid syntax`

Pull from master: utils.py in tacotron 2 has only 29 lines and your error is on line 31.


Monthly Competitive Coding Challenges (Online / Offline)

Discussion regarding the monthly coding challenges can be done here. Have to decide the platform, duration, competition type, etc. @bkhanale  Prateek Pattnaik @sr6033  @nawedx .

Please add  Prateek Pattnaik to the org I don't know his github handle

Monthly coding events can be done on online platforms like hackerearth codechef etc and for onsite events like swig and code and principium we can host them on our judge that we are currently developing it will be custom made to suit our requirements.

@bkhanale can you set up campus chapters for Hackerrank or codechef maybe we can get them as sponsors for Advaita if we maintain active chapters.

We can arrange a Tech-Talk with Anup Kalbalia (Tech lead, CodeChef) if we want next year.

Yes please @bkhanale that would be great

Can anyone ask Swapnil Sourabh and Abhishek Patel to accept the invites and become active here we need all the help that we can.

@bkhanale  Keep a watch at all other opportunities regarding competitive coding events, so that we can get max. incentives and resources as well as outreach.

There would be few demands from their side as well. They basically need us to cover the travelling expenses and accommodation costs. Though if he had planned to come to Bhubaneswar at that time then the costs would reduce.

That part can be taken care by the Tech Club funds and all.

That part can be taken care by the Tech Club funds and all.

We have those? Do we ever get to use them?

@bkhanale @palash25 Inviting Anup Kalbalia would be great. We can also think about inviting Gaurav Sen. He has recently given Tech Talks on competitive programming in some NIT's.

See  #24

I wanted to bring Anup this semester only but we fell short of budget. Make sure to keep a separate fund either from ACM or TECH for speakers from the beginning.

I wanted to bring Anup this semester only but we fell short of budget.

For what do we have to spend money on? Is it airfare? Can't these speakers just buy their own flight tickets? I mean they are all working professionals.

There are some terms these speakers follows , we have to talk to them before hand about that, as far as it is possible , hospitality wont be a problem from tech club side but travelling budget is something to think about.

I'm all for hospitality but I'm not in favor of calling speakers who work for well-established startups and large conglomerates but ask students to pay for their airfare. Its okay if these speakers are students themselves then we can think about paying for their tickets.
But that's just me sharing my opinion and if you guys unanimously decide to call a speaker and fund for their air travel I won't have any problem with that.

@palash25 can you please ,merge the speaker discussion to one issue only. Kind of staggered ho raha hai.

Actually there is no need to merge the issue we just need to stop talking about tech talks on this thread. Not sure who started and deviated us from this issue, I've lost all context.

Update : The Coding Challenge for the month of May , successfully conducted. Codechef HEAT JAM May 2018

Can we have the competitive coding challenges every fortnight?
This will encourage people to learn and practice more. Also, it will help students to gain confidence regarding their coding skills which in turn will help them participate in more coding contests online.

Every fortnight is a good idea but we need more people to come forward to manage it, right now Bhushan and Prateek are the problem setters, if we will plan for every fortnight we can have then on Hackerearth and Codechef alternatively ,but again the same thing more people to come forward and take responsibility . Also we can allow students from other colleges and keep it open. More thoughts required on this.

@arijeetsat @bkhanale make a list of potential problem setters. And then we can alternatively ask them to set questions for the events.

@sonakshisaxena1 Conducting contests on our own every fortnight would not be feasible. The most we can do is one contest per month. I have the list of problem setters with me, but certainly for getting one contest per month we will need more problem setters. Also, since Codeforces and Codechef have many short contests we can always promote them for participation. They have some quality problems and editorials as well.

Okay one thing, what is our main purpose of holding contests, lets be clear abt then the frequency matters.
As per me, we are holding them to encourage our students to participate in local contest ,compete against each other , also to maintain a CC culture.
Thoughts please??

I agree with @bkhanale on this. You cannot maintain this kind of frequency of contests we just don't have that many setters.
Also after a while, CP newbies will move onto real competitions like Long challenges and the popularity of these challenges will go down.

Monthly with good quality questions and more participation  is what we should focus on maintaing.👍👍

@arijeetsat Yes, but just not competing but improve themselves, learn new concepts. We need more good programmers for competitions like CodeJam, Hackercup, ACM ICPC. And the Codechef long and Codeforces rounds do good job in learning part. To improve participation we need the local contests.

One monthly challenge by the club and we can give some goodies to good performers in other challenges going on in codechef, hackerrank, codeforces etc etc,

Okay. I get the issue of setting up the problems.
But as @bkhanale said that there are many short contests available in Codechef/Codeforces, then can we gather up people and encourage them to give any selected contest together, in CG or some other rooms?
This will stop them to procrastinate about giving a contest someday and will also help them to discuss solutions and editorials together. They might be able to form a good team for such contests like ICPC etc in future too.

help them to discuss solutions and editorials together. They might be able to form a good team for such contests like ICPC etc in future too.

This is something that has to be done, discussion on monthly challenges on the major platforms , at least Codechef, if not all. Also, for own competition, we also need to have editorials.


feat: Add listener rules support for http_tcp

Description
Adding the http_tcp_listener_rules variable defining rules for non https ports.
It works the same as https_listener_rules on #155.
Doc updated with the pre-commit hook.
Motivation and Context
We need to test rules creation on listeners before having SSL certificates implemented.
Since https_listener_rules was the only way to do this, it was not yet possible to do the same for testing.
Breaking Changes
No breaking change since it is a new feature.
How Has This Been Tested?
Tested using one of my company project, we created an ALB and some rules to forward traffic by using this new variable. Just used a temporary fork of this project with these changes.
Also updated the example to have some basic tests.
Ran pre-commit checks to ensure formatting and terraform validate pass.
Closes #214
Ref #176

Hello,
Is anybody available for reviewing this ?

v6.4.0 has been just released.


v1.3.0 exited before replying to getmanifest

Until version 1.2.1 I have been enjoying this plugin without issues. Upgrading to 1.3.0 throws the following error when starting:
$ lightning-cli plugin start summars/summars
{
   "code": -3,
   "message": "/home/lightning/summars-v1.3.0/summars: exited before replying to getmanifest"
}

$ lightning-cli getinfo
"version": "v23.08.1"

Thats very unusual.

Did you compile it yourself or used the release binaries?
Can you provide more logs from lightningd?

Also this command: lightning-cli plugin start summars/summars does not seem correct when looking at the response where the actual binary is. Can you try lightning-cli plugin start /home/lightning/summars-v1.3.0/summars ?

I just found the issue. I was trying to run the release binaries and seems that 1.3.0 needs glibc >=2.32 and this machine is still running debian oldstable which uses glibc 2.31.
Time for me to stop delaying the upgrade to bookworm.
/summars-v1.3.0/summars: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/lightning/summars-v1.3.>
/summars-v1.3.0/summars: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/lightning/summars-v1.3.>
/summars-v1.3.0/summars: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /home/lightning/summars-v1.3.>

Thank you for the quick reply and for the great plugin.

ah forgot about that, i should add a hint for that requirement when using release binaries. You should still be able to build it yourself though. Thanks for the feedback!


Why images stay temporary folder when using multiple image upload editor?

If I use ImageUploadEditor image file moves related (EntityName/000/000/xyxyxyx.png) folder with renaming. But MultipleImageUploadEditor attribute puts images to temporary folder and stay there even after save entity? I dont use any parameter, there is only attrbute name
[MultipleImageUploadEditor]
public String Resimler { get; set; }
Serenity ver: 1.9

It's probably a bug, will have a look.

Can't reproduce this. They don't stay after save entity. They only stay if you select some file and close without saving, which works as designed. They got deleted after some time, if you have a ".temporary" in temporary folder.

I could. If I put MultipleImageUploadEditor to top of form property this issue occurs. But If I put MultipleImageUploadEditor to top of Row field works good. You can ty and fix if necessary you think. Thanks.

Yes if you put it to Form, repository has no idea there is a MultipleImageUploadEditor related to that field so MultipleImageUploadBehavior can't handle uploaded images, so not only files will stay in temporary folder, they won't be uploaded / processed at all. So it has to be at row.

I use MultipleFileUploadEditor in Row.cs with attr FilenameFormat = "IndividualTraining/Certificate/~"
but file could not move to IndividualTraining/Certificate folder even following folder not created.


Should use code block for the actual command

in https://developer.fermyon.com/spin/quickstart

the command rustup target add wasm32-wasi should be inside code block

This particular example has been fixed in #207 — any other examples where this needs to be addressed?

See linear.app's Developer Docs team for more information.


time given in seconds

The time for the running submissions is given in seconds. However, if this time is very large it becomes not very informative, eg

It would be better to have something of type h : m : s in the display

+1000

+1. Last year in #276 I rounded it to second, but @rth had a remark that for very short jobs we should have an integer 0. So some more sophisticated rounding would be perhaps better. In money-prize challenges we usually give them a max total time given in minutes, so the rounding should not be smaller than minutes, but we could get rid of the seconds if it's more than, say, 1h.


Fix incorrect prism dimension

Codecov Report

Merging #12 (ae09b6e) into main (978f949) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main      #12   +/-   ##
=======================================
  Coverage   62.24%   62.24%
=======================================
  Files          10       10
  Lines         686      686
=======================================
  Hits          427      427
  Misses        259      259

Impacted Files
Coverage Δ

src/corrsum_based/correlationsum_boxassisted.jl
84.11% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


A sentence in Basic Image Classification Tutorial is not clear

URL(s) with the issue:
Please provide a link to the documentation entry, for example:
https://www.tensorflow.org/tutorials/keras/classification#make_predictions
Description of issue (what needs changing):
The statement, The model's linear outputs, logits. can't be understood.
More explanation can be added on why the Outputs are Linear.

In that blog, the outputs of the model are of course linear. They define the model as
model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10) # < ---------------- Look here
])

Here you can see they didn't use any activation and by default, it's None. And that makes it a linear layer. So the output would be logit. And further, they add another layer which is softmax to get probabilities from the logit.
probability_model = tf.keras.Sequential([model, tf.keras.layers.Softmax()])

So, model here is giving linear outputs (logit) but when we add the softmax layer then it will give probabilities from the logit.

@innat,
Thank you for your response. I agree with your explanation but my point is that we can add the same explanation in the Tutorial so that it will be clear to Newbies. Also, the sentence formation of the statement, The model's linear outputs, logits, is not proper and is incomplete.

It is a image classification problem that is why we are using Softmax()  activation function in
probability_model = tf.keras.Sequential([model, tf.keras.layers.Softmax()])

@worldpeaceaspirer logits are normally raw outputs of a classification model. A Normalization function like softmax is added for normalized probabilities in the prediction. And the logits hyperlink clearly explains what logits are and how we can use normalization functions based on the type of classification problem. Beginners normally have an idea of logits and the use of activation functions so I guess there is no need to add anything new to the tutorial.

Reviewed. Thanks everyone!


Behavior of first line indentation in paragraphs seems limiting

As I gathered from the docs (as seen on the reference for the first-line-indent argument), the first paragraph on a page is special-cased to never have it's first line indented.
If I understand this correctly, that means not all paragraphs will have indentation applied, just like LaTeX does by default. If this is correct, then some styles will not be able to be represented with Typst. For example, in Brazilian Portuguese documents we are used to apply indentation to every paragraph, regardless of it's length or position in the text, and as such we use the LaTeX package indentfirst pervasively.
Did I understand it correctly?

We'll add some configuration for this, so that both ways are possible.

Possibly related, paragraphs starting with something other than unformatted text don’t get indented (e.g. #super[x]). All of this makes the feature feel temperamental.

The bug with unformatted text not being indented should be fixed since the last release. Can you still reproduce it? More generally, I tried to fix the first line indent, but its not as simple as it seems. As currently all text is layouted as a paragraph in Typst, removing the rule would indent everything. That includes headings, code blocks, captions, words in boxes, etc. Since paragraphs are not clearly demarked (as they would be in, say HTML), we first need some automatic way to distinguish these two kinds of texts (possibly through the context they appear in).

this problem is still exist

Yes, I know. As mentioned in my previous response, the overall issue is difficult to fix. The only thing that was fixed so far is that formatted text wasn't even indented when it wasn't in the first paragraph.

I found another para hanging-indent
if you want to indent first line in 1 em
set par(justify: false, leading: 1.5em, first-line-indent:2em )

you can do like this:
set par( hanging-indent:-1em )

@DawnEver oh wow, I never thought if using a negative hanging indent, great idea! Though if you have a border around the text you'd have to adjust the the block's padding so it wouldn't clip out of it. However it looks like you got around that by setting the first-line-indent to be 2em already! I'll try it out sometime, thanks for sharing :)

I found another para hanging-indent if you want to indent first line in 1 em
set par(justify: false, leading: 1.5em, first-line-indent:2em )

you can do like this:
set par( hanging-indent:-1em )

This works anyway. But it will also affect the raw texts/code blocks.

This one also does the trick (thanks https://github.com/shuosc/SHU-Bachelor-Thesis-Typst/issues/12#issuecomment-1506872937)
show heading: it =>  {
    it
    par()[#text(size:0.5em)[#h(0.0em)]]
}

This one also does the trick (thanks shuosc/SHU-Bachelor-Thesis-Typst#12 (comment))
show heading: it =>  {
    it
    par()[#text(size:0.5em)[#h(0.0em)]]
}

so quick and so dirty, but I like it 👍

@laurmaedje sorry for bothering, but do you know if there are plans to resolve this?

My temporary solution:

@teenjuna There are definitely plans to resolve it. Before we can do it, we have to have a stronger model of what is a paragraph and what is just text. That requires some deeper changes to how Typst works internally. The same changes are also relevant for HTML export. It will still take some time, but it is still on my radar.

@laurmaedje thanks for response! Good to know this is also related to HTML renderer, as it insures that this issue won't stale.
My 2 cents: I would expect Typst to have a separate Text element that just holds text. It would be used inside lists, equations, etc. Paragraph would then me just a container for top-level text on a page and can have styles that won't affect just Text.
In HTML, such Text element would map to , which is not semantically incorrect according to MDN (as span element doesn't have special meaning).

in Brazilian Portuguese documents we are used to apply indentation to every paragraph

Same goes with Russian.

show heading: it =>  {
    it
    par()[#text(size:0.5em)[#h(0.0em)]]
}

This can be optimized like so:
show heading: it => {
  it
  par(text(size: 0.5em, "")) // using [] instead of "" isn't the same
}

But this introduces additional vertical spacing, which isn't a desired result.

Surprisingly (to me), this works perfectly with no issues (maybe there are some):
#let indent = 1.25cm // Russian standard AFAIK
#set par(first-line-indent: indent)
#show heading: it => { it; h(indent) }

or
#let indent = 1.25cm
#set par(first-line-indent: indent)
#show heading: it => {
  it
  h(indent)
}

#let annotation(body)={
par(
  [#h(3em)#body]
)
}

This is the safest and easiest approach for now. Here is a simplified version:
#let indent = 1.25cm
#set par(first-line-indent: indent)

#let indent_par(body) = par(h(indent) + body)
// or
#let indent_par(body) = par[#h(indent)#body]

Then just do:
= Heading
#indent_par[any text or #lorem(30)] // First paragraph

#lorem(30) // nth paragraph
...

// or
= Heading
#indent_par(lorem(30) // First paragraph

#lorem(30) // nth paragraph
...

I found another para hanging-indent if you want to indent first line in 1 em
set par(justify: false, leading: 1.5em, first-line-indent:2em )

you can do like this:
set par( hanging-indent:-1em )

Maybe it worked in previous versions, but now (0.7) it doesn't (text goes out of the default margins):

source code
#set par(first-line-indent: 2em)

= Heading
#lorem(30)

#lorem(30)

#lorem(30)

#set par(first-line-indent: 2em, hanging-indent: -1em)

= Heading
#lorem(30)

#lorem(30)

#lorem(30)

Maybe some additional settings are necessary.

One other way is to set a show rule, but you can't undo in v0.7:
#show "|": h(indent) // |
#show ">": h(indent) // >
#show "~": h(indent) // \~

You can choose almost any Unicode character, but I think that letters are easier to use (since it's the intended way of using the show rule). I really would like to use some fancy syntax for show rules, but you can't do that right now. And regex("") is therefore mostly useless.

Maybe add a note to the Paragraph documentation that the following workarounds can be used to get the desired behavior?
I suggest adding these three
// Workaround 1: Manually add horizontal space
#let indent = 3em // or whatever value you want
#let indent_par(body) = par[#h(indent)#body]
// Usage:
#indent_par[First paragraph]

// Workaround 2: Special character
#show "¬": h(3em) // Use whatever symbol and space you want
// Usage:
¬First paragraph

// Workaround 3: Automatically add empty paragraph after heading
#show heading: it => {
    it
    par(text(size:0.35em, h(0.0em)))
} // Only works for paragraphs directly after heading

And what did this "unfix" then?

IDK, i don't understand what consecutive means. And here seems to be some explaination https://github.com/typst/typst/issues/2350#issuecomment-1754422551

Maybe @PgBiel or @laurmaedje can explain. I, too, don't understand what's the point of consecutive right now.

Consecutive is what indents only the second and following paragraphs. Just removing it won't have the intended effect though because suddenly everything would be indented: Headings, captions, single worlds in boxes, etc.
Try writing A #box[B] C in your fixed version. B will also be indented.
The fundamental problem we need to fix first is that text can't exist outside of a paragraph at the moment.

The fundamental problem we need to fix first is that text can't exist outside of a paragraph at the moment.

Let's clear something here. Can text exist outside of a paragraph in Typst 0.9?
If I understand correctly, this statement must be true eventually:

text can't exist outside of a paragraph

In all Typst versions to date, text will always be layouted as part of a paragraph. In the future, we have to define some separation because not all text should be affected by paragraph indent.

In the future, we have to define some separation

"Some separation" means a new element type or something? Or is this the main problem: to define "some separation" exactly?

That's the main problem.

In all Typst versions to date, text will always be layouted as part of a paragraph

So any text in any mode will be wrapped in a par element, correct? If I explicitly wrap text in a par element, will it be wrapped twice or the implicit par element will be substituted by the explicit par element?
In some documents, text can be used just as a single word inserted in a specific place/position, i.e., not as a text paragraph. Therefore, such example shouldn't be affected by indent, hence shouldn't implicitly be wrapped in par element. This also refers to heading, box element and probably much more.
This makes me think that the only 2 instances where there is a paragraph (that need to be indented) are these:
Just a simple and small paragraph in markup mode.

#par[Another paragraph that is wrapped in explicit `par` element.]

Wouldn't this fix the problem? Only apply implicit par to a text in markup or if the par used explicitly. In any other circumstances, par won't be used, and no indent will be applied. Sounds to me like a case closed, just need to fix some logic in Rust. What am I missing here?
Or in the current version of Typst, you can't physically "not use par" because it is hard-coded in every single element? If so, then par and its use must be re-thinked and re-done (re-written).

Just a simple and small paragraph in markup mode.

#par[Another paragraph that is wrapped in explicit `par` element.]

One problem that I see here is inline insertion of something into the pure text paragraph, specifically at the very beginning of the paragraph. I think that if this something is not another element or does not contain another element (referring to content), then we make it a part of a pure text paragraph (mixed paragraph). Meaning that in such case, indent will be present. But if this something is another element or contain such, then indent won't be present.
Here are some examples (where → is Typst-created paragraph indentation):
→Just a simple and small paragraph in markup mode.
→Just a simple and #string small #integer paragraph #array-with-some-simple-type-values in markup mode.
→Just a simple and small #grid([`block` element splits 1 paragraph into 2])→paragraph in markup mode.
→#[content without elements] Just a simple and small paragraph in #content-value markup mode.
#box[explicit non-`par` element removes indent/implicit `par`]Just a simple and small paragraph in markup mode.
→#par[Just a simple and small paragraph in markup mode.]
→#par[This will probably create a separate paragraph and move text on the right to the 2nd implicit paragraph (like it is in Typst 0.9).]→Just a simple and small paragraph in markup mode.
→#{
  for index in range(5) {
    str(index)
  }
} Just a simple and small paragraph in markup mode.
#table([table is `block`, so text on the right will go to a new line, therefore a new paragraph.])→Just a simple and small paragraph in markup mode.
→Just a simple and #box(table([inline table])) small paragraph in markup mode.

The examples above just show how I see the logic work in practice if this issue (#311) is fixed, and I think all of them obey my simple rules stated above:
Add indent (make a par element) if:

pure text in root markup mode;
par is explicitly used to wrap "paragraph text";
text with inline (box) elements not at the beginning of a paragraph (not at the 1st character);
text with inline content (no blocks inside) not at the beginning of a paragraph (not at the 1st character);
text with block (not inline) elements not at the beginning of a paragraph (not at the 1st character), BUT such elements will be put on a newline and all the text after these elements will be also put on another newline, therefore creating a new par;
text with simple type values at the beginning of a paragraph (at the 1st character).

I'm not sure about inline elements at the beginning of a paragraph, like this one:
#box[explicit non-`par` element removes indent/implicit `par`]Just a simple and small paragraph in markup mode.

Maybe this should automatically be converted to par. I can imagine an (inline) image/icon/custom bullet at the beginning of a paragraph, so the indent should be present in such cases.

I'm not sure about inline elements at the beginning of a paragraph, like this one:
#box[explicit non-`par` element removes indent/implicit `par`]Just a simple and small paragraph in markup mode.

My intuition is that inline content at the start of a paragraph should not have any affect on indentation. After all, that's what it means to be inline: it's inside the paragraph—it just happens to be at the beginning. What if I wanted to start my paragraph off with a little icon or logo, for whatever reason?
If you wanted to remove implicit indentation before inline content, you would just use the exact same method that applied to any other paragraph.
#let noindent = h(0pt, weak: true)

#noindent Some paragraph completely on its own.

#noindent #box[A paragraph that] has inline content at its start.

If you wanted to remove implicit indentation before inline content, you would just use the exact same method that applied to any other paragraph

Sometime it would be very handy, but in some situations simple #h(-5em) doesn't work. There should be a forceful way (function) to always be able to remove (any) indentation. Another example is enumerated list (can't remove indent without breaking the numeration).

Maybe straying a bit into "general discussion" territory here, but something I come across quite frequently, that Andrew also touched on, is putting block-level things within the flow of a paragraph.
For example, my only real gripe with Typst so far has been managing spacing around lists or enums that are between paragraphs:
Some paragraph that uses a list to introduce:
- One topic;
- Two topics; or
- Maybe even more topics!
And then concludes with a few more words down here.

As far as layout is concerned, these are three block-level elements: par, list, par. But, as far as writing is concerned, it contains one topic and is one single paragraph. I would expect the second par to have no extra indentation, and have regular line-spacing between it and the list (at least when the list is tight).
I have so far been unable to get Typst to reliably do this without putting v(0pt) spaces below tight lists, to consume the paragraph's above. That has the problem that now I can't end a paragraph with a list, since now it will eat up the spacing between what should be the next paragraph (not that that's all too proper, grammatically speaking, but I have been using Typst for not-necessarily-formal things like assignments). I think Typst should make it both possible and easy to achieve that sort of spacing, if I so desire.
HTML and CSS have this same problem, too: given a sequence of just <p>, <ul>, <p> elements, it's impossible to know if the second <p> is the continuation of the first "paragraph," or the start of a following one. That is, without putting a no-indent class on the second one, or something.
In Typst, however, whitespace is significant: an empty line means something. Perhaps Typst could make the distinction by whether or not a blank line appears between the last list item and the following paragraph:
This paragraph starts here...
- And goes into a list...
Before returning to a paragraph, without indentation. Then, it
- Goes back into a list...
- And ends here (however informal that may be).

Because there's an empty line here, _this_ is a *whole new paragraph,* instead of a continuation of the previous one.

That syntax seems a bit... loose, though. Just bringing up a concern of mine, and throwing an idea out there. 😁

Sometime it would be very handy, but in some situations simple #h(-5em) doesn't work. There should be a forceful way (function) to always be able to remove (any) indentation. Another example is enumerated list (can't remove indent without breaking the numeration).

h should work for that with the weak parameter, I think...

https://typst.app/docs/reference/layout/h/#parameters-weak
Weak spacing in markup also causes all adjacent markup spaces to be removed, regardless of the amount of spacing inserted. To force a space next to weak spacing, you can explicitly write #" " (for a normal space) or ~ (for a non-breaking space). The latter can be useful to create a construct that always attaches to the preceding word with one non-breaking space, independently of whether a markup space existed in front or not.

That should "remove (any) indentation," shouldn't it? Or does it not work on paragraph indentation? 🤔

@matthew-e-brown Before the list, this is already special-cased. Not at the bottom though. I think back then I figured that it only makes sense to consider a tight list at the end of a paragraph and continuing after that would be strange. Maybe it was the wrong call.

Ok, maybe I went too far with this. Generally, using a negative indentation with #h() does give the desired result, but it should be rather an #undo-indent function, because I don't want to either create a #let undo-indent = h(-indent-value) shortcut myself or use a bare (and ugly) #h(-indent-value) all the time. Because I really do want Typst to be super readable (that's kinda the point when comparing with LaTeX). But I did face some problems. This example works pretty good:
source

#set par(first-line-indent: 1em)
#set list(body-indent: 1em)

First paragraph.

Second paragraph.

#h(-1em)Third paragraph.

- item

  #h(-1em)Paragraph

  #h(-1em)#box(table(columns: 2, [cell], [cell]))

  Also need to subtract a length of "#sym.bullet" somehow (I can _add_ by using
  ```typ #hide(sym.bullet)```, which isn't ideal either) to make a perfect 0-level
  root indentation (no indentation):

  #h(-2em)#box(table(columns: 2, [cell], [cell]))
- item

output

But here is a bug (cell content is not indented by the same value/distance from all 4 sides inside the cell):
source

#set par(first-line-indent: 1em)
#set list(body-indent: 1em)

- item

  #h(-2em)#box(table(columns: 2, [cell], [cell]))

output

I have another example, but I currently can't show it. Maybe later.

Here it is. I now figured out that I didn't add an empty line after the list item, so the table wasn't able to dedent.
source

+ word
  #h(-1em)#box(table(
    columns: (2fr, 2fr, 1fr),
    [a], [b], [c],
    ..range(3 * 7).map(_ => [~]),
  ))
+ word

output

With an empty line:
source

+ word

  #h(-1em)#box(table(
    columns: (2fr, 2fr, 1fr),
    [a], [b], [c],
    ..range(3 * 7).map(_ => [~]),
  ))
+ word

output

Hi @araruna, @matthew-e-brown, and others,
As a work-around, I suggest the following code to custom first-line indent via some conditions:
#show: body => {
  for (ie, elem) in body.children.enumerate() {
    if elem.func() == text {
      if ie > 0 and body.children.at(ie - 1).func() == parbreak {
        h(2em)
      }
      elem
    } else {
      elem
    }
  }
}

And the following content
= Section

#lorem(40)
+ abc
+ def
+ ghi
#lorem(20)
- asd
- wds
- wasd

#lorem(20)

== Subsection

will result in

But I think it would be better if there is a parameter that we can control whether the first line in the immediately folllowing paragraph, not the paragraph after a parbreak, is indented.

One more workaround that works for headings, lists, figures, tables etc.
let indent_hack = it => {
        show par: set block(spacing: 0em)
        it; v(-1.15em); ""
}

set par(justify: true, first-line-indent: 2em)
show heading: it => {
    indent_hack(it)
    v(0.3em)
}
show list: indent_hack
show enum: indent_hack
show table: indent_hack
show grid: indent_hack
show rect: indent_hack
show figure: indent_hack

# Add other display functions after which line should be with indent

Hey, friends. This has been an interesting issue, so I'll try to explain what's happening.
Paragraphs only contain "inline" elements. This includes
content.is::<SpaceElem>()
            || content.is::<TextElem>()
            || content.is::<HElem>()
            || content.is::<LinebreakElem>()
            || content.is::<SmartQuoteElem>()
            || content.to_packed::<EquationElem>()
                .is_some_and(|elem| !elem.block(styles))
            || content.is::<BoxElem>()

(Text, horizontal spacing, linebreaks, smart quotes, inline equations, and boxes).
This means, if any element other than one of those shows up, it isn't included in a paragraph. Suppose you have two paragraphs and you want to put a block equation in the first one. Before the block equation is put in, Typst simply has two paragraphs. After the equation is put in, however, typst splits the first paragraph into two: one before the equation and one after the equation. Now, there are essentially four blocks:

par1(1/2)
eq
par1(2/2)
par(2)
This also happens for lists, headers, anything that isn't one of the elements I've listed above.

Now, on top of this, a paragraph is only indented if it follows another paragraph. This is the consecutive: bool variable. If a paragraph follows another paragraph, consecutive = true. Therefore, using any item other than the ones I listed splits the paragraph in two, the second of which will not be indented because it doesn't technically follow another paragraph.
It's quite easy to add an argument always-indent-first-line: bool to the paragraph item, which is something I've done in my forked repo. However, this means that an indent will occur on everything that Typst considers a paragraph. For example,
Consider a manifold, $cal(M)$, and the ordinary differential equation,// This is its own paragraph, so it's indented.
$
  (d x) / (d t) = u (x(t))
$
where // This is its own paragraph, so it's indented.
$
  u: cal(M) -> T cal(M)
$
is a vector field on the manifold. // This is its own paragraph, so it's indented.

Even though all of this is one contiguous thought, the way Typst lays this out is odd to the everyday English writer. My proposal is to allow non-inline elements to be included in paragraphs. I'm working on modifying that in my fork.
There are several other issues that are related to this. Here is one:
https://github.com/typst/typst/issues/3206#issue-2083875484

Hi @araruna, @matthew-e-brown, and others,
As a work-around, I suggest the following code to custom first-line indent via some conditions:
#show: body => {
  for (ie, elem) in body.children.enumerate() {
    if elem.func() == text {
      if ie > 0 and body.children.at(ie - 1).func() == parbreak {
        h(2em)
      }
      elem
    } else {
      elem
    }
  }
}

And the following content
= Section

#lorem(40)
+ abc
+ def
+ ghi
#lorem(20)
- asd
- wds
- wasd

#lorem(20)

== Subsection

will result in

But I think it would be better if there is a parameter that we can control whether the first line in the immediately folllowing paragraph, not the paragraph after a parbreak, is indented.

This word around is super amazing but I would like to add that the indentation would not come into effect if you are using some styling options like bold. See the screenshot for a toy example. But simply, add #h(2em)  before the not-indented paragraph can do a good job.
= Example heading

Some text here.

#h(2em) _Here_: some bold text at start.

Hi @Fanisting,
The issue you mentioned occurs because the leading element is strong , not text. So the indent condition elem.func() == text and ie > 0 and body.children.at(ie - 1).func() == parbreak cannot be true in this case. To fix this issue, a better indent condition can be applied to cover this special case.

I started a new project which is a good reason to finally try and stop using 〜 as a manual intent after every heading, list, figure etc. and instead test workarounds that were posted here. Maybe I missed something, but the only good one was posted by @HPDell, because other workarounds mess up the layout with vertical or horizontal spacing.
I initially used it as is, and it was already very nice to have automatic indentation everywhere. But soon I faced my first problem: when inserting medium [] it is not flattened and therefore adds a nested structure ("subdocument"). But thankfully by adding 2 lines of code I was able to fix it using a recursive show rule function.
And soon enough, some places started misbehaving. In the end, one place where I can test several situations started to work as expected no matter what I did. But I still don't have anything besides paragraphs and headings, so my current solution will probably break again in the future. But for now this looks robust enough:
snippet
#let indent = 1em
#let styled = [#set text(red)].func()
#let space = [ ].func()
#let sequence = [].func()

#let turn-on-first-line-indentation(
  doc,
  last-is-heading: false, // space and parbreak are ignored
  indent-already-added: false,
) = {
  for (i, elem) in doc.children.enumerate() {
    let element = elem.func()
    if element == text {
      let previous-elem = doc.children.at(i - 1)
      if i == 0 or last-is-heading or previous-elem.func() == parbreak {
        if not indent-already-added {
          indent-already-added = true
          h(indent)
        }
      }
      elem
    } else if element == heading {
      indent-already-added = false
      last-is-heading = true
      elem
    } else if element == space {
      elem
    } else if element == parbreak {
      indent-already-added = false
      elem
    } else if element == sequence {
      turn-on-first-line-indentation(
        elem,
        last-is-heading: last-is-heading,
        indent-already-added: indent-already-added,
      )
    } else if element == styled {
      styled(turn-on-first-line-indentation(
        elem.child,
        last-is-heading: last-is-heading,
        indent-already-added: indent-already-added,
      ), elem.styles)
    } else {
      indent-already-added = false
      last-is-heading = false
      elem
    }
  }
}

#show: turn-on-first-line-indentation

P.S. There is little logic to it: I was just checking the output and trying to make a bunch of conditions, so that there will be just one indentation at a single place in a document (paragraph).

Hi @Andrew15-5 ,
Thanks for your workaround which seems powerful and is able to cover more situations.
After posting my solution, I was thinking that is there any possibility to set up an array of conditions and check their results with functions like any() and all() to avoid too many if-else structure. So that users can add their own conditions to control the indent behaviour.

The outer if conditions are all for element type checking, which means that you probably can add only maybe 15 more, which is a lot, but not infinite (and most wouldn't be big anyway). To combine multiple elements under one condition, you can do if element in (table, figure, raw) {}. I tried making a switch case thingy, but it was far less readable.

I'm pretty sure in some standards/countries not having a first-line indentation is the correct way.

I'm pretty sure I have never seen the indent missing on a new page.

I found this package that someone made to workaround this problem: https://github.com/flaribbit/indenta. It seems to work pretty well.

This simple trick #{ "" } works for me:
#set par(first-line-indent: 1.5em, justify: true, leading: 1.5em)

== First one

// This make the first line of the first paragraph to be indented
#{ "" }

#lorem(200)

#lorem(200)

This and many other "tricks" and "workarounds" create an additional vertical spacing/indentation that can be unacceptable for some people (if you have strict guidelines, or you're a perfectionist). And if you're still going to use it, then you can also use #"" instead.
More about adding vertical spacing (near headings) rather than horizontal indentation, but there is still some overlap.
That's why I now only use root show rule for this one. You have a full control over what is being inserted and where (https://github.com/typst/typst/issues/311#issuecomment-2104447655). The only problem I found with a few of my designs, is that I can't trace the state of the variables after a nested sequence. I think that it's fixable by returning state/variables in addition to the element, but I haven't tried it. This means that the current design doesn't work with nested blocks of content, a.k.a. a file with #include statement(s). Which in turn forced me to use a flat file structure, otherwise I will have incorrect spacing near headings where #includes connect.

#show heading: it => [
#it.numbering
#it.body
#{""}
]

This simple trick #{ "" } works for me:
#set par(first-line-indent: 1.5em, justify: true, leading: 1.5em)

== First one

// This make the first line of the first paragraph to be indented
#{ "" }

#lorem(200)

#lorem(200)

Following your solution, this fixes works :
#show heading: it => [
  #it
  #{""}
]

For the perfectionists, add a negative vertical spacing after creating a new paragraph. I do not like this because it reminds me of the LaTeX quick fixes (gambiarras in my language), which I am trying to avoid using Typst.
show heading: it => {
    set text(12pt)
    it
    ""
    v(-1em)
}

Personally I prefer h(1em) for paragraphs and pad(left: 1em) for lists.
#set par(first-line-indent: 1em, justify: true, leading: 1em)

= lorem

#lorem(20)

#lorem(20)

#pad(left: 1em, [
  + #lorem(20)
  + #lorem(20)
])

#h(1em)#lorem(20)

#lorem(20)

just add show heading: el => el + ""

Let me add my two cents of a workaround as well
#show heading: it => [ #it #v(-par.spacing) #{ "" } ]
#show figure: it => [ #it #v(-par.spacing) #{ "" } ]
#show terms: it => [ #it #v(-par.spacing) #{ "" } ]
#show list: it => [ #it #v(-par.spacing) #{ "" } ]

Let me add my two cents of a workaround as well

For future reference, I've posted a similar solution in the forum as well, with examples: https://forum.typst.app/t/how-to-indent-every-paragraph-excluding-headings/2137/4
It boils down to:
// Add more elements below as necessary, e.g. list
#show selector.or(heading, table, grid, figure): it => {
    it
    ""
    context v(-par.spacing - measure("").height)
}

It's worth noting that you can unify the show rules by using selector.or as above. Also, for consistency with the original heading spacing, you should exclude the spacing for "".

Are there any known or potential issues with this particular method? Perhaps if some set of settings is changed, meaning the state is nothing like default settings? I'm thinking of a fuzzy testing of sorts. All because it's so simple yet so powerful/robust (unlike all others), at least on the paper. I'll put it to the test with my next documents.

Let me add my two cents of a workaround as well
#show heading: it => [ #it #v(-par.spacing) #{ "" } ]
#show figure: it => [ #it #v(-par.spacing) #{ "" } ]
#show terms: it => [ #it #v(-par.spacing) #{ "" } ]
#show list: it => [ #it #v(-par.spacing) #{ "" } ]

I had to wrap the figure in a #block[] (#box[] prevents breakable tables) because large image can create an extra blank page.

I found that it doesn't work after #pagebreak():
#let indent = 1.25cm
#set list(indent: indent)
#set par(first-line-indent: indent)
#show heading: set align(center)
#let complex-title-page(title: none) = align(center + horizon)[#title]

#show selector.or(heading, list, enum, figure, page): it => {
  it
  ""
  context v(-par.spacing - measure("").height)
}

#complex-title-page(title: "My title")
#pagebreak()

#lorem(20)

#lorem(20)

- #lorem(5);
- #lorem(5);
- #lorem(5).

#lorem(20)

== Heading
#lorem(20)
#figure(rect(), caption: [])
#lorem(20)

Correct; in general, it requires something to be before the paragraph.
It isn't very easy to solve that without a native implementation. But I think it's OK to manually handle those specific cases for the time being.

But I think it's OK to manually handle those specific cases for the time being.

#show "〜": h(indent) is like:


Speed up checking for existing files by vectorizing it?

Right now with keep_existing=True we check for an existing file at the prospective output location right before we would process that file. It might be possible to save time by checking for existing files for all input files up front.

With skip_func, the user can create a set of the existing files up front and then check against that set.


[SPARK-3325][Streaming] Add a parameter to the method print in class DStream.

I have deleted the repository with mistake, so I create a new PR. old PR:https://github.com/apache/spark/pull/2216
def print(num: Int = 10)

User can control the number of elements which to print.

Oops sorry, my bad. I created another merge conflict by merging another PR. Let me fix that for you.

Oops sorry, my bad. I created another merge conflict by merging another PR. Let me fix that for you.

Test build #24978 has finished for   PR 3237 at commit bb35d1a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #24978 has finished for   PR 3237 at commit bb35d1a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24978/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24978/
Test PASSed.

I opened a new PR with your branch
https://github.com/apache/spark/pull/3865
You will get the main authorship :)

I opened a new PR with your branch
https://github.com/apache/spark/pull/3865
You will get the main authorship :)

The other PR #3865 has been merged. Mind closing this PR? Thanks for all the effort!

Mind closing this PR?