straight is rebuilding org on every single launch of emacs

What's wrong
I am now seeing a Building org... message from straight.el every single time I launch emacs.  Presumably this is related to the recent addition of support for :build steps.  Since the build steps do things like compiling the whole manual, this takes something like one or two minutes and therefore drastically increases the waiting time before emacs is ready for use.
Directions to reproduce

Launch emacs with a config which uses straight.el to load org ;-)

I've added some debug to the bit of straight-use-package which calculates modified and it is showing that (straight--package-might-be-modified-p recipe no-build) returns 98, so presumably it's something in there which is going wrong.
But if you can't reproduce this and it's looking like this might be related to some peculiarity of my config, given that I have a decent amount of elisp hacking experience, rather than trying to create a minimal test case I guess it might be quicker if you can confirm I'm on the right track and I'll keep digging in straight--package-might-be-modified-p.
Version information

Latest master of straight.el
Emacs version: 27.1
Operating system: openSUSE Tumbleweed

Try running the following test case twice.
It will build straight/Org in a clean environment in /tmp/org-rebuild-test.straight on the first execution.
Once that finishes, you should see something similar to the first Output below.
Then run it again. It should run much faster (hopefully Org is not being rebuilt) the second time and the Output should eb similar to the second Output block below.
Test Case
(straight-bug-report
  :user-dir "org-rebuild-test.straight"
  :preserve t
  :post-bootstrap
  (straight-use-package 'org))

Test run at: 2020-11-01 16:16:52
system-type: gnu/linux
straight-version: prerelease (HEAD -> feat/el-get-recipe-support, fork/feat/el-get-recipe-support) 9aa6141 2020-10-31
emacs-version: GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.17.3, Xaw3d scroll bars) of 2020-10-31

Output
```emacs-lisp
Bootstrapping straight.el...
Bootstrapping straight.el...done
Rebuilding all packages due to build cache schema change
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done
Test run with version: prerelease (HEAD -> develop, origin/develop) 68dade0 2020-10-24
Cloning org...
Cloning org...done
Building org...
Building org...done
</details>

<details><summary>Output</summary>

```emacs-lisp
Test run with version: prerelease (HEAD -> develop, origin/develop) 68dade0 2020-10-24

Thanks a lot for the pointers!  I'm unsure if you meant to run within an existing emacs or a fresh emacs -Q or something else?  If I just eval-defun that sexp then I only see:
Testing straight.el in directory: /tmp/org-rebuild-test.straight

and nothing else happens.
OTOH if you meant with emacs -Q or similar then I am unsure how the bootstrapping is supposed to work because of course straight-bug-report is not defined yet.

Sorry, I should have been clearer.
You'll want to run that in a regular Emacs session. It will start another Emacs process that runs the test. When it finishes, a buffer with the test's output (*straight-bug-report-process*) should pop up with the results. The test will install straight and Org, so it should take about as long as it does on your normal system.

If you're not seeing anything after a reasonable amount of time, you can add to the test to make it an interactive emacs process:
(straight-bug-report
  :interactive t
  :user-dir "org-rebuild-test.straight"
  :preserve t
  :post-bootstrap
  (straight-use-package 'org))

You can also check to see if /tmp/org-rebuild-test.straight exists on disk.

@progfolio commented on November 2, 2020 7:37 PM:

Sorry, I should have been clearer.
You'll want to run that in a regular Emacs session.

That's what I did.  It just outputs the message above in the minibuffer and nothing else happens...  Ahah, eventually I saw this in the minibuffer:
error in process sentinel: unless: Symbol’s value as variable is void: interactive

It happened so much later (minutes later) that I didn't realise it was related.  But running it again showed the connection.  After enabling debug-on-error I get this:
Debugger entered--Lisp error: (void-variable interactive)
  (if interactive nil (unless raw (straight-bug-report--format report)) (switch-to-buffer-other-window straight-bug-report--process-buffer))
  (unless interactive (unless raw (straight-bug-report--format report)) (switch-to-buffer-other-window straight-bug-report--process-buffer))
  (lambda (_process _event) (unless interactive (unless raw (straight-bug-report--format report)) (switch-to-buffer-other-window straight-bug-report--process-buffer)) (unless preserve-files (when (file-exists-p temp-emacs-dir) (delete-directory temp-emacs-dir 'recursive))))(#<process *straight-bug-report-process*> "finished\n")

Sounds like you may be executing it without lexical binding enabled. I patched this last night so that you'll get a more informative user-error in this case. Try executing in a buffer which has lexical-binding enabled.

Ahah yes! Works now, many thanks :-)  Yes, on second run it doesn't rebuild org.  So does that mean the issue is specific to my environment?  If so, what's the next step - my original suggestion of digging deeper in straight--package-might-be-modified-p?

So does that mean the issue is specific to my environment?

Yes, that's my inclination. Test case runs for me without subsequent rebuilds.
Stepping through straight--package-might-be-modified-p could be a good lead.
Some other useful info to have:

value of straight-check-for-modifications
recipe for org in init file/ recipe for Org as returned by straight-get-recipe.

Thanks a lot - super busy at the moment but I'll get back to this soon!

I have been having similar issues with org-contrib. I have resorted to delete all content in straight/ but it still crops up. I'll follow the suggestions here too and see what happens. Interestingly doing a manual eval from withing my init file (C-x C-e outside use-package form) stopped this for a while. It's very annoying and very confusing ;)

@rileyrg please try running the test case here in a buffer with lexical binding enabled:
https://github.com/raxod502/straight.el/issues/624#issuecomment-720153249
Any other relevant info, such as your org use package declaration and the values I mentioned here:
https://github.com/raxod502/straight.el/issues/624#issuecomment-720750877
will be helpful as well.

@aspiers I noticed you are using use-package's :ensure keyword alongside the :straight keyword:
https://github.com/aspiers/emacs/blob/9eeee3e99e5e9129bb63a16e7fe81452bd0025bc/.emacs.d/init.d/as-org-mode.el#L8
This could cause a conflict w straight since :ensure utilizes package.el. Does removing that help?

Nice idea but it doesn't help.  However @rileyrg's screenshot above made me wonder why I was seeing Building org... rather than Building org-plus-contrib... so I looked closer and actually saw both in the *Messages* buffer, but the former is the one which takes a long time.
This suggests to me that maybe this issue is related to the way I'm loading org-plus-contrib.

Hrm, I changed some stuff and now both builds are taking a long time ... :-/
If I have
(use-package org
  :straight org-plus-contrib
  ...)

does that mean that I should be referencing org or org-plus-contrib in clauses such as this :after section?
(use-package orgit
  :after (org-plus-contrib magit))

@progfolio commented on November 2, 2020 10:06 PM:

Some other useful info to have:

value of straight-check-for-modifications

The default: (find-at-startup find-when-checking)

recipe for Org in your init file and recipe for Org as returned by straight-get-recipe.

org or org-plus-contrib?  In any case, here they both are:
`(org :type git :repo "https://code.orgmode.org/bzg/org-mode.git" :local-repo "org" :build ,(let ((make (if (eq system-type 'berkeley-unix) "gmake" "make")) (emacs (concat "EMACS=" invocation-directory invocation-name))) `(,make "oldorg" ,emacs)))

`(org-plus-contrib :type git :repo "https://code.orgmode.org/bzg/org-mode.git" :local-repo "org" :files (:defaults "contrib/lisp/*.el") :build ,(let ((make (if (eq system-type 'berkeley-unix) "gmake" "make")) (emacs (concat "EMACS=" invocation-directory invocation-name))) `(,make "oldorg" ,emacs)))

output of (gethash "org-plus-contrib" straight--build-cache) (or "org" if you're not using "org-plus-contrib")

("2020-11-08 11:21:04" nil (:type git :repo "https://code.orgmode.org/bzg/org-mode.git" :local-repo "org" :files (:defaults "contrib/lisp/*.el") :build ("make" "oldorg" "EMACS=/usr/bin/emacs") :package "org-plus-contrib"))

@aspiers commented on November 8, 2020 11:19 AM:

Hrm, I changed some stuff and now both builds are taking a long time ... :-/
If I have
(use-package org
  :straight org-plus-contrib
  ...)

does that mean that I should be referencing org or org-plus-contrib in clauses such as this :after section?
(use-package orgit
  :after (org-plus-contrib magit))

Oh wow, I changed all of these from org-plus-contrib to org and now the constant rebuilding is gone!

I guess there are two remaining questions:

@rileyrg does this help you fix it?
Are there any improvements which could be made to the docs in this regard?

I'm. Also having constant issues with boxquote.el btw. Keeps saying can't
run git. I'm only on my mobile atm so just consider this as a "you're not
alonec support. The strangest thing to me re org contrib is that C-x C-e on
the use package builds it and then it's not rebuilt on subsequent restarts
as we'd hope.
On Sun, 8 Nov 2020, 12:44 Adam Spiers<EMAIL_ADDRESS>wrote:

@progfolio https://github.com/progfolio commented on November 2, 2020
10:06 PM
https://github.com/raxod502/straight.el/issues/624#issuecomment-720750877
:
Some other useful info to have:

value of straight-check-for-modifications

The default: (find-at-startup find-when-checking)

recipe for Org in your init file and recipe for Org as returned by
straight-get-recipe.

org or org-plus-contrib? In any case, here they both are:
(org :type git :repo "https://code.orgmode.org/bzg/org-mode.git" :local-repo "org" :build ,(let ((make (if (eq system-type 'berkeley-unix) "gmake" "make")) (emacs (concat "EMACS=" invocation-directory invocation-name))) (,make "oldorg" ,emacs)))
(org-plus-contrib :type git :repo "https://code.orgmode.org/bzg/org-mode.git" :local-repo "org" :files (:defaults "contrib/lisp/*.el") :build ,(let ((make (if (eq system-type 'berkeley-unix) "gmake" "make")) (emacs (concat "EMACS=" invocation-directory invocation-name))) (,make "oldorg" ,emacs)))

output of (gethash "org-plus-contrib" straight--build-cache) (or
"org" if you're not using "org-plus-contrib")

("2020-11-08 11:21:04" nil (:type git :repo "https://code.orgmode.org/bzg/org-mode.git" :local-repo "org" :files (:defaults "contrib/lisp/*.el") :build ("make" "oldorg" "EMACS=/usr/bin/emacs") :package "org-plus-contrib"))
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raxod502/straight.el/issues/624#issuecomment-723566007,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYTC6R3CLGSQB7PF34JMDSO2ACTANCNFSM4TGXVASQ
.

This issue describes the problem where it is rebuilt, undesirably, on subsequent restarts.  Are you saying you want the opposite behaviour?  It's not supposed to rebuild unless something changes which requires the rebuild.

This issue describes the problem where it is rebuilt, undesirably, on subsequent restarts. Are you saying you want the opposite behaviour? It's not supposed to rebuild unless something changes which requires the rebuild.

I am saying that it was constantly rebuilding up to when I started emacs again, watched it rebuild org-plus-contrib yet again, opened my config file and manually eval'ed the org use-package form. Then, when I quit and restarted emacs it didn't rebuild. Strange eh? But it might trigger an "aha" in someone who knows this stuff better than me.

FYI, I have to give up now because of other things today, but I did a complete straight wipe out and restarted emacs. I had to comment out the boxquote.el form. When I tried to evaulate it after here is the form and stack:-
     (use-package boxquote
       :bind
       ("C-S-r" . boxquote-region))

Debugger entered--Lisp error: (error "Failed to run \"git\"; see buffer *straight-process*")   signal(error ("Failed to run \"git\"; see buffer *straight-process*"))   error("Failed to run %S; see buffer %s" "git" "*straight-process*")   straight--get-call-raw("git" "clone" "--origin" "origin" "--branch" "master" "--no-checkout" "git@github.com:davep/boxquote.el.git" "/home/rgr/.emacs.d/straight/repos/boxquote.el/")   apply(straight--get-call-raw "git" ("clone" "--origin" "origin" "--branch" "master" "--no-checkout" "git@github.com:davep/boxquote.el.git" "/home/rgr/.emacs.d/straight/repos/boxquote.el/"))   straight--get-call("git" "clone" "--origin" "origin" "--branch" "master" "--no-checkout" "git@github.com:davep/boxquote.el.git" "/home/rgr/.emacs.d/straight/repos/boxquote.el/")   straight-vc-git--clone-internal(:depth full :remote "origin" :url "git@github.com:davep/boxquote.el.git" :repo-dir "/home/rgr/.emacs.d/straight/repos/boxquote.el/" :branch "master" :commit nil)   straight-vc-git-clone((:type git :flavor melpa :host github :repo "davep/boxquote.el" :package "boxquote" :local-repo "boxquote.el") nil)   apply(straight-vc-git-clone ((:type git :flavor melpa :host github :repo "davep/boxquote.el" :package "boxquote" :local-repo "boxquote.el") nil))   straight-vc(clone git (:type git :flavor melpa :host github :repo "davep/boxquote.el" :package "boxquote" :local-repo "boxquote.el") nil)   straight-vc-clone((:type git :flavor melpa :host github :repo "davep/boxquote.el" :package "boxquote" :local-repo "boxquote.el"))   straight--clone-repository((:type git :flavor melpa :host github :repo "davep/boxquote.el" :package "boxquote" :local-repo "boxquote.el") nil)   #f(compiled-function () #<bytecode 0x157cc17d2bb5>)()   straight--transaction-exec(use-package-\(:type\ git\ :flavor\ melpa\ :host\ github\ :repo\ \"davep/boxquote\.el\"\ :package\ \"boxquote\"\ :local-repo\ \"boxquote\.el\"\)-nil-nil :now #f(compiled-function () #<bytecode 0x157cc17d2bb5>))   straight-use-package(boxquote)   (progn (straight-use-package 'boxquote) (defvar use-package--warning101 #'(lambda (keyword err) (let ((msg (format "%s/%s: %s" ... keyword ...))) (display-warning 'use-package msg :error)))) (condition-case err (progn (if (fboundp 'boxquote-region) nil (autoload #'boxquote-region "boxquote" nil t)) (let* ((name "C-S-r") (key (if (vectorp name) name (read-kbd-macro name))) (kmap (if (and nil ...) (symbol-value nil) nil)) (kdesc (cons (if ... name ...) (if ... nil ...))) (binding (lookup-key (or kmap global-map) key))) (let ((entry (assoc kdesc personal-keybindings)) (details (list ... ...))) (if entry (setcdr entry details) (add-to-list 'personal-keybindings (cons kdesc details)))) (define-key (or kmap global-map) key #'boxquote-region))) ((debug error) (funcall use-package--warning101 :catch err))))   eval((progn (straight-use-package 'boxquote) (defvar use-package--warning101 #'(lambda (keyword err) (let ((msg ...)) (display-warning 'use-package msg :error)))) (condition-case err (progn (if (fboundp 'boxquote-region) nil (autoload #'boxquote-region "boxquote" nil t)) (let* ((name "C-S-r") (key (if ... name ...)) (kmap (if ... ... nil)) (kdesc (cons ... ...)) (binding (lookup-key ... key))) (let ((entry ...) (details ...)) (if entry (setcdr entry details) (add-to-list ... ...))) (define-key (or kmap global-map) key #'boxquote-region))) ((debug error) (funcall use-package--warning101 :catch err)))) nil)   elisp--eval-last-sexp(nil)   eval-last-sexp(nil)   funcall-interactively(eval-last-sexp nil)   call-interactively(eval-last-sexp nil nil)   command-execute(eval-last-sexp)

That error simply means the git clone failed, which could be down to a broken internet connection or some other misconfiguration of your git setup.  I suggest manually trying the git clone command listed in the error to see if it's even related to emacs.

The clone works fine converting to
git clone https://github.com/davep/boxquote.el
indicating it's something (recent) with my ssh/gnupg agent in this case.
I'll change the recipe. The only one in the hundred or so packages I use :(
On Sun, 8 Nov 2020, 14:31 Adam Spiers<EMAIL_ADDRESS>wrote:

That error simply means the git clone failed, which could be down to a
broken internet connection or some other misconfiguration of your git
setup. I suggest manually trying the git clone command listed in the
error to see if it's even related to emacs.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raxod502/straight.el/issues/624#issuecomment-723577739,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYTC2LNSAQD26CNLUNXOTSO2MT3ANCNFSM4TGXVASQ
.

I just did a clean start ensuring emacs was opening my key properly for ssh-agent. the git@github  boxquote.el clone still failed from inside emacs. I'm removing it now so I can get on. Sorry I cant provide more info atm.

@rileyrg: I get the git failure as well. The error being thrown points to the *straight-process* buffer which reveals:
Cloning into '/home/n/.emacs.d/straight/repos/boxquote.el'...
fatal: Remote branch master not found in upstream origin

It appears boxquote.el's default branch is now "main":
https://github.com/davep/boxquote.el
either of these should fix the problem while we wait for #543 to land:
(straight-use-package '(boxquote :branch "main"))

or
(use-package boxquote
  :straight (:branch "main"))

@aspiers: Thank you for taking the time to look into this more and for #628. Glad to see you were able to resolve the issue. I agree we could do better by providing more specific configuration examples in the docs for popular packages like Org. If you have ideas, please let us know.

Urgh, actually I'm not sure I have resolved this.  The problem with changing them all from org-plus-contrib to org is that I think straight is actually now building both, but presumably I only want it to build org-plus-contrib.   I agree that #620 will help by preventing the risk of this duplication, but it won't necessarily solve any underlying problem with unnecessary rebuilding.

Here's a test I used:
(use-package org-bullets
  :straight t
  :after org-plus-contrib    ;; N.B. *not* just org here
  :config
  (require 'org-contacts)

(use-package org
  :straight org-plus-contrib)

org-contacts.el is only in Org's contrib/lisp so this tests whether org-plus-contrib is being correctly loaded.  If the :after is changed to org or commented out, the require will fail.

Does moving the use-package declaration for org before the declaration for org-bullets help?

Yes but the whole point of using :after was to avoid dependencies on lexical ordering as my emacs config is split across many files.

I understand. I just wanted to make sure we were at the root cause of the issue. The semantics of use-package's :after keyword only affect loading. Even if we take straight out of the equation and just :ensure a package, it will still be installed
prior to it being loaded by an :after declaration:
(use-package a
  :ensure t
  :after b)

expands to:
(progn
  (use-package-ensure-elpa 'a
                           '(t)
                           'nil)
  (eval-after-load 'b
    '(require 'a nil nil)))

Similarly, with :straight t (or any variation) the expansion is:
(progn
  (straight-use-package 'a)
  (eval-after-load 'b
    '(require 'a nil nil)))

You could install org-plus-contrib earlier in your init and then
configure it in a separate declaration/file by either having two use-package forms,
or by directly calling (straight-use-package 'org-plus-contrib) before installing packages which would transitively load Org.
I do something similar to the latter in my config.

Sorry chaps, I've been out of it.  This main/master solution I'll try later
today. Thanks. It seems progress bring made
On Tue, 10 Nov 2020, 05:24 Nicholas Vollmer<EMAIL_ADDRESS>wrote:

I understand. I just wanted to make sure we were at the root cause of the
issue. The semantics of use-package's :after keyword only affect loading.
Even if we take straight out of the equation and just :ensure a package,
it will still be installed
prior to it being loaded by an :after declaration:
(use-package a
:ensure t
:after b)
expands to:
(progn
(use-package-ensure-elpa 'a
'(t)
'nil)
(eval-after-load 'b
'(require 'a nil nil)))
Similarly, with :straight t (or any variation) the expansion is:
(progn
(straight-use-package 'a)
(eval-after-load 'b
'(require 'a nil nil)))
You could install org-plus-contrib earlier in your init and then
configure it in a separate declaration/file by either having two
use-package forms,
or by directly calling (straight-use-package 'org-plus-contrib) before
installing packages which would transitively load Org.
I do something similar to the latter in my config.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raxod502/straight.el/issues/624#issuecomment-724441252,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYTC6H2KDR2TCLWXLSQ2TSPC57LANCNFSM4TGXVASQ
.

@rileyrg please try running the test case here in a buffer with lexical binding enabled:
#624 (comment)
Any other relevant info, such as your org use-package declaration and the values I mentioned here:
#624 (comment)
will be helpful as well.

I
(use-package org
        :straight org-plus-contrib
        :custom
        (org-directory "~/.emacs.d/orgfiles")
        (org-agenda-files (list org-directory (concat org-directory "/projects" )))
        (org-refile-targets `((,(directory-files-recursively org-directory "^[[:alnum:]].*\\.\\(org\\|gpg\\)\\'") :maxlevel . 9)))
        (org-outline-path-complete-in-steps nil)         ; Refile in a single go
        (org-refile-use-outline-path t)                  ; Show full paths for refiling
        :config
        (require 'org-protocol)
        (require 'org-inlinetask)
        (defun rgr/org-agenda (&optional arg)
          (interactive "P")
          (let ((org-agenda-tag-filter-preset '("-trash")))
            (org-agenda arg "a")))
        :bind
        ("C-c a" . org-agenda)
        ("C-c A" . rgr/org-agenda)
        ("C-c c" . org-capture)
        ("C-c l" . org-store-link)
        ("C-c C-l" . org-insert-link)
        ("C-c C-s" . org-schedule)
        ("C-c t" . org-todo))

Output:-

Test Case
(straight-bug-report
  :user-dir "org-rebuild-test.straight"
  :preserve t
  :post-bootstrap
  (straight-use-package 'org))

Test run at: 2020-11-10 10:22:25
system-type: gnu/linux
straight-version: prerelease (HEAD -> develop, origin/develop) 0f283e2 2020-11-04
emacs-version: GNU Emacs 27.1 (build 12, x86_64-pc-linux-gnu, GTK+ Version 3.24.23) of 2020-09-15

Output
Bootstrapping straight.el...
Bootstrapping straight.el...done
Rebuilding all packages due to build cache schema change
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> develop, origin/develop) 0f283e2 2020-11-04
Cloning org...
Cloning org...done
Building org...
Building org...done

output of (gethash "org-plus-contrib" straight--build-cache):-

("2020-11-10 10:13:43" nil (:type git :repo "https://code.orgmode.org/bzg/org-mode.git" :local-repo "org" :files (:defaults "contrib/lisp/*.el") :build ("make" "oldorg" "EMACS=/usr/local/bin/emacs") :package "org-plus-contrib"))

straight-check-for-modifications is a variable defined in straight.el.

Value
(find-at-startup find-when-checking)
Original Value
(find-at-startup find-when-checking)

@rileyrg Thanks for running that. Try running it again and sharing the output. (If you've restarted since the first run or emptied your temporary file directory, you will have to run it two more times). That way we can see if Org is being built on a subsequent startup or if it is a matter of configuration.

@rileyrg Thanks for running that. Try running it again and sharing the output. (If you've restarted since the first run or emptied your temporary file directory, you will have to run it two more times). That way we can see if Org is being built on a subsequent startup or if it is a matter of configuration.

Two straight runs (pun intended ;)):-

Test Case
(straight-bug-report
  :user-dir "org-rebuild-test.straight"
  :preserve t
  :post-bootstrap
  (straight-use-package 'org))

Test run at: 2020-11-11 10:24:59
system-type: gnu/linux
straight-version: prerelease (HEAD -> develop, origin/develop) 0f283e2 2020-11-04
emacs-version: GNU Emacs 27.1 (build 12, x86_64-pc-linux-gnu, GTK+ Version 3.24.23) of 2020-09-15

Output
Bootstrapping straight.el...
Bootstrapping straight.el...done
Rebuilding all packages due to build cache schema change
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> develop, origin/develop) f7ff754 2020-11-11
Cloning org...
Cloning org...done
Building org...
Building org...done

Test Case
(straight-bug-report
  :user-dir "org-rebuild-test.straight"
  :preserve t
  :post-bootstrap
  (straight-use-package 'org))

Test run at: 2020-11-11 10:26:04
system-type: gnu/linux
straight-version: prerelease (HEAD -> develop, origin/develop) 0f283e2 2020-11-04
emacs-version: GNU Emacs 27.1 (build 12, x86_64-pc-linux-gnu, GTK+ Version 3.24.23) of 2020-09-15

Output
Test run with version: prerelease (HEAD -> develop, origin/develop) f7ff754 2020-11-11

I should mention though that hand evaling my org form AFER emacs has fully loaded and rebuilt org-plus-contrib solves the issue for subsequent restarts too.

Looks like the test only builds Org on the first start.
You can add this to the start of your config:
  (eval-after-load "org" '(debug))

and launch emacs with emacs --debug-init.
You should hit the debugger immediately after Org is loaded.
It would be useful to see when/where it is being loaded.

@rileyrg :
Looks like the test only builds Org on the first start.
You can add this to the start of your config:

  (eval-after-load "org" '(debug))

and launch emacs with emacs --debug-init.
You should hit the debugger immediately after Org is loaded.
It would be useful to see when/where it is being loaded.

Debugger entered: nil
(lambda nil (debug))()
funcall((lambda nil (debug)))
mapc(funcall ((lambda nil (debug))))
do-after-load-evaluation("/home/rgr/.emacs.d/straight/build/org-plus-contrib...")
load-with-code-conversion("/home/rgr/.emacs.d/straight/build/org-plus-contrib..." "/home/rgr/.emacs.d/straight/build/org-plus-contrib..." nil t)
require(org)
apply(require org)
byte-compile-file-form-require((require 'org))
byte-compile-file-form((require 'org))
#f(compiled-function (form) #<bytecode 0x15613a213919>)((require 'org))
byte-compile-recurse-toplevel((require 'org) #f(compiled-function (form) #<bytecode 0x15613a213919>))
byte-compile-toplevel-file-form((require 'org))
#f(compiled-function (inbuffer) #<bytecode 0x15613a2ee365>)(#<buffer  Compiler Input>)
byte-compile-from-buffer(#<buffer  Compiler Input>)
byte-compile-file("/home/rgr/.emacs.d/straight/build/org-plus-contrib..." nil)
byte-recompile-file("/home/rgr/.emacs.d/straight/build/org-plus-contrib..." force 0)
#f(compiled-function () #<bytecode 0x15613a551f9d>)()
byte-recompile-directory("/home/rgr/.emacs.d/straight/build/org-plus-contrib..." 0 force)
straight--byte-compile-package((:type git :repo "https://code.orgmode.org/bzg/org-mode.git" :local-repo "org" :files (:defaults "contrib/lisp/.el") :build ("make" "oldorg" "EMACS=/usr/local/bin/emacs") :package "org-plus-contrib"))
straight--build-package((:type git :repo "https://code.orgmode.org/bzg/org-mode.git" :local-repo "org" :files (:defaults "contrib/lisp/.el") :build ("make" "oldorg" "EMACS=/usr/local/bin/emacs") :package "org-plus-contrib") nil)
#f(compiled-function () #<bytecode 0x15613a30eac9>)()
straight--transaction-exec(use-package-(:type\ git\ :repo\ "https://code.orgmode.org/bzg/org-mode.git"\ :local-repo\ "org"\ :files\ (:defaults\ "contrib/lisp/*.el")\ :build\ ("make"\ "oldorg"\ "EMACS=/usr/local/bin/emacs")\ :package\ "org-plus-contrib")-nil-nil :now #f(compiled-function () #<bytecode 0x15613a30eac9>))
straight-use-package(org-plus-contrib)
eval-buffer(#<buffer  load-584805> nil "/home/rgr/.emacs.d/config/init.el" nil t)  ; Reading at buffer position 460
load-with-code-conversion("/home/rgr/.emacs.d/config/init.el" "/home/rgr/.emacs.d/config/init.el" nil nil)
load("/home/rgr/.emacs.d/config/init.el")
eval-buffer(#<buffer  load> nil "/home/rgr/.emacs.d/init.el" nil t)  ; Reading at buffer position 128
load-with-code-conversion("/home/rgr/.emacs.d/init.el" "/home/rgr/.emacs.d/init.el" t t)
load("/home/rgr/.emacs.d/init" noerror nomessage)
startup--load-user-init-file(#f(compiled-function () #<bytecode 0x15613a326285>) #f(compiled-function () #<bytecode 0x15613a326299>) t)
command-line()
normal-top-level()

Did adding the :branch to the boxquote recipe fix the other issue for you?

Yes,partly. But there was another issue - and I dont have the time to look into it today. It was the only one not working if my gpg agent was failing. Frustrating. It only started happening recently so I'll file this away as a debian startup issue for now.

@rileyrg I cloned your emacs.d and played around to see what I could find. I'm fairly certain the issue is in the init-load config/init.el. You're calling org-babel-load-file which I believe is where the mix up is. I opened a pull with a fix that works locally for me. See if that fixes the rebuilding issue for you.

I'll try that this morning. Just an "fyi" this rebuilding business is quite
new with "nothing" changed on my side.
On Thu, 12 Nov 2020 at 08:10, Nicholas Vollmer<EMAIL_ADDRESS>wrote:

@rileyrg https://github.com/rileyrg I cloned your emacs.d and played
around to see what I could find. I'm fairly certain the issue is in the
init-load config/init.el. You're calling org-babel-load-file which I
believe is where the mix up is. I opened a pull with a fix that works
locally for me. See if that fixes the rebuilding issue for you.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raxod502/straight.el/issues/624#issuecomment-725886581,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYTCZGZH47EN3MNZ3BKYDSPOC7LANCNFSM4TGXVASQ
.

@rileyrg I cloned your emacs.d and played around to see what I could find. I'm fairly certain the issue is in the init-load procedure in config/init.el. You're calling org-babel-load-file which I believe is where the mix up is. I opened a pull with a fix that works locally for me. See if that fixes the rebuilding issue for you.

Sorry to report , it's still rebuilding. At first, I thought it was because I'd forgotten to recreate my.elc. I pulled it into my "master" btw so you can see it there as I thought you'd like a quick turn around as opposed to me figuring out how to test it in a different branch (something I need to learn to do).

There are a lot of layers of indirection in your .emacs.d set up. Would you mind walking me through how it is supposed to work?
You mention an init.elc, but you're only ever loading init.el as far as I can tell.

The init.elc I mentioned was the one in the config/ directory. Funnily
enough, I just renamed it emacs-config-init-el to avoid confusion with the
top-level .emacs.d/init.el which emacs "auto finds". The top-level
just loads the "real" init in config/  since time and time again emacs
wrote over the symlink in .emacs.d directory when I merely symlinked
config/init to .emacs.d/init. I have no doubt it can be simplified or
cleaned up.
On Thu, 12 Nov 2020 at 09:20, Nicholas Vollmer<EMAIL_ADDRESS>wrote:

There are a lot of layers of indirection in your .emacs.d set up. Would
you mind walking me through how it is supposed to work?
You mention an init.elc, but you're only ever loading init.el as far as I
can tell.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raxod502/straight.el/issues/624#issuecomment-725919096,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYTCZ2SXT6WYMKTHTQVJ3SPOLDTANCNFSM4TGXVASQ
.

I would argue that byte-compiling that file won't provide any meaningful performance gains either.

I'm into the wee-hours of Eastern Time, so I've got to get to bed, but I'm 99% certain this is a configuration issue now. I'd be glad to help you sort it out and look over your emacs configuration if you'd like. I think simplifying a bit would make things easier to reason about and likely see a faster init time. Let me know and we can move the discussion to a separate issue on your repo.

I agree re byte-compiling and it's why I had the .emacs.d/init.el load the .emacs.d/config/init.el specifically now I think about it.  Nice of you to offer. Let me run through it and tidy it if possible first as I don't want to burden you. But I'm not the only one with this rebuild issue I think.
re walkthrough, there's only one "redirection" and thats that .emacs.d/init.el exists to load the (now renamed) .emacs.d/config/init.el (emacs-config-init.el with my next pus). Why? Because sometimes with a broken config emacs would wipe the .emacs.d/init.el symlink to .emacs.d/config/init.el file. Maybe it doesnt do it anymore. History etc.

I've been having the same issue, I used the following configuration in the past which worked.
(use-package org
  :straight (org-plus-contrib
             :repo "https://code.orgmode.org/bzg/org-mode.git"
             :local-repo "org"
             :files (:defaults "contrib/lisp/*.el")
             :includes (org)))

Switching to configuration which doesn't specify the repository url doesn't help.
(use-package org :straight org-plus-contrib)

How should org-mode be included now?

@jdek: The comment you've linked to shows a proposed :includes keyword, but that has not been implemented.
Please run the test case in this comment twice in a buffer with lexical binding enabled: https://github.com/raxod502/straight.el/issues/624#issuecomment-720153249

@progfolio Ah, I didn't realise it was a no-op. I think my case may have been related to being on an older Emacs (26.1 I think). I've just updated to 27.1 and the following seems to work:
(use-package org
  :straight (org-plus-contrib
             :repo "https://code.orgmode.org/bzg/org-mode.git"
             :local-repo "org"
             :files (:defaults "contrib/lisp/*.el"))

Thanks for the help.

Test Case
(straight-bug-report
  :user-dir "org-rebuild-test.straight"
  :preserve t
  :post-bootstrap
  (straight-use-package 'org))

Test run at: 2020-11-18 01:39:45
system-type: darwin
straight-version: prerelease (HEAD, origin/develop) 7b5a930 %cs
emacs-version: GNU Emacs 27.1 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1671.60 Version 10.14.6 (Build 18G95)) of 2020-08-12

Output
Bootstrapping straight.el...
Bootstrapping straight.el...done
Rebuilding all packages due to build cache schema change
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> develop, origin/develop) 7b5a930 %cs
Cloning org...
Cloning org...done
Building org...
Building org...done

Test Case
(straight-bug-report
  :user-dir "org-rebuild-test.straight"
  :preserve t
  :post-bootstrap
  (straight-use-package 'org))

Test run at: 2020-11-18 01:40:54
system-type: darwin
straight-version: prerelease (HEAD, origin/develop) 7b5a930 %cs
emacs-version: GNU Emacs 27.1 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1671.60 Version 10.14.6 (Build 18G95)) of 2020-08-12

Output
Test run with version: prerelease (HEAD -> develop, origin/develop) 7b5a930 %cs

The output from the test runs looks good: no rebuild on the second run. Glad you got it working. Just out of curiosity, can I see the value of M-x straight-get-recipe org-plus-contrib?

Output of M-x straight-get-recipe org-plus-contrib
`(org-plus-contrib
 :type git
 :repo "https://code.orgmode.org/bzg/org-mode.git"
 :local-repo "org"
 :files (:defaults "contrib/lisp/*.el")
 :build ,(let ((make (if (eq system-type 'berkeley-unix) "gmake" "make"))
               (emacs (concat "EMACS=" invocation-directory invocation-name)))
           `(,make "oldorg" ,emacs)))

Thank you

I have this issue again. A rm -rf ~/.emacs.d/straight/{build,build-cache.el,repos} sometimes fixed it. There is no visible issues again using the test-cases (apart from the first test-case failing for some reason). On macOS, exec-path-from-shell is required to correct the PATH to include the package manager's makeinfo command as the built-in version is too old to build org-mode.
 If I then switch to my config from the minimal config in-place without deleting anything then org-mode is not rebuilt. I assume this means something in my config is touching org-mode's files timestamps but only sometimes? (also only org-mode nothing else is ever rebuilt)
Test cases
Test Case 1
(straight-bug-report
  :user-dir "org-rebuild-test.straight"
  :preserve t
  :post-bootstrap
  (straight-use-package 'org))

Test run at: 2020-11-20 23:01:40
system-type: darwin
straight-version: prerelease (HEAD, origin/develop) 7b5a930 %cs
emacs-version: GNU Emacs 27.1 (build 1, x86_64-apple-darwin, NS appkit-2022.10 Version 11.0.1 (Build 20B29)) of 2020-11-18

Output
Bootstrapping straight.el...done
Bootstrapping straight.el...
Bootstrapping straight.el...done
Rebuilding all packages due to build cache schema change
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> develop, origin/develop) 7b5a930 %cs
Cloning org...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> develop, origin/develop) 7b5a930 %cs
Building org...

Failed to run "make": make: *** No rule to make target `oldorg'.  Stop.
Warning (straight): Failed to run "make"; see buffer *straight-process*
Building org...done

Test Case 2
(straight-bug-report
  :user-dir "org-rebuild-test.straight"
  :preserve t
  :post-bootstrap
  (straight-use-package 'org))

Test run at: 2020-11-20 23:03:15
system-type: darwin
straight-version: prerelease (HEAD, origin/develop) 7b5a930 %cs
emacs-version: GNU Emacs 27.1 (build 1, x86_64-apple-darwin, NS appkit-2022.10 Version 11.0.1 (Build 20B29)) of 2020-11-18

Output
Test run with version: prerelease (HEAD -> develop, origin/develop) 7b5a930 %cs
Building org...
Building org...done

Test Case 3
(straight-bug-report
  :user-dir "org-rebuild-test.straight"
  :preserve t
  :post-bootstrap
  (straight-use-package 'org))

Test run at: 2020-11-20 23:08:29
system-type: darwin
straight-version: prerelease (HEAD, origin/develop) 7b5a930 %cs
emacs-version: GNU Emacs 27.1 (build 1, x86_64-apple-darwin, NS appkit-2022.10 Version 11.0.1 (Build 20B29)) of 2020-11-18

Output
Test run with version: prerelease (HEAD -> develop, origin/develop) 7b5a930 %cs

@jdek: Sorry to hear that. It most likely is a configuration error. If you would like to link me to your config, I'd be happy to take a look at it.

Appreciate everyone's patience. This is a thorny one. Here's what I'm thinking is at the root of all this:

byte-compiling a package which requires Org effectively requires it in the installation environment. I wasn't aware of this, but it looks like it has been mentioned before and implementing #72 should take care of this.
straight's default recipes for Org should probably specify :no-build t and :no-autoloads t by default, as the :build step takes care of these and is most likely what is causing the org-loaddefs.el changed on disk prompt and is redundant work.
possible third issue with a package which has already executed :build being rebuilt when it is a dependency of another package (@jdek: I'm seeing this now with your config with org-journal declaring Org as a dependency.) Though, I think there's a chance #72 could resolve this as well.

I'll implement #72 first and keep everyone posted. Thanks again for your patience and the information you've provided.

Looks like you have a plan of action on this @progfolio, sharing an additional test case below in case it's helpful.
Also it seems like some people have resolved when testing with straight-bug-report only to still experience the issue with their config.  I found that if I changed the test case code without using a fresh :user-dir it seemed to trick straight-bug-report.  But as soon as I ran the same test case twice in a fresh :user-dir the problem would present it self again.
Not sure what the cause is but thought it might be an explanation why the problem seems to go away when testing.

Test Case
(straight-bug-report
  :user-dir "org-contrib-rebuild-test.straight"
  :preserve t
  :post-bootstrap
  (straight-use-package 'use-package)
  (use-package org :straight
    (org-plus-contrib :repo "https://code.orgmode.org/bzg/org-mode.git" :local-repo "org" :files
      (:defaults "contrib/lisp/*.el")))
  (use-package org-super-agenda :straight t :after org))

Test run at: 2020-12-16 09:16:41
system-type: darwin
straight-version: prerelease (HEAD -> develop, origin/develop) 0f283e2 2020-11-04
emacs-version: GNU Emacs 28.0.50 (build 1, x86_64-apple-darwin19.6.0, NS appkit-1894.60 Version 10.15.7 (Build 19H2)) of 2020-12-12

First Run Output
Bootstrapping straight.el...
Bootstrapping straight.el...done
Rebuilding all packages due to build cache schema change
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> develop, origin/develop) 949a0a7 2020-12-13
Cloning use-package...
Cloning use-package...done
Building use-package...
Building use-package → Building bind-key...
Building use-package → Building bind-key...done
Building use-package...
Building use-package...done
Cloning org (for org-plus-contrib)...
Cloning org (for org-plus-contrib)...done
Building org-plus-contrib...
Building org-plus-contrib...done
Cloning org-super-agenda...
Cloning org-super-agenda...done
Building org-super-agenda...
Building org-super-agenda → Cloning s.el...
Building org-super-agenda → Cloning s.el...done
Building org-super-agenda → Building s...
Building org-super-agenda → Building s...done
Building org-super-agenda → Cloning dash.el...
Building org-super-agenda → Cloning dash.el...done
Building org-super-agenda → Building dash...
Building org-super-agenda → Building dash...done
Building org-super-agenda → Building org...
Building org-super-agenda → Building org...done
Building org-super-agenda → Cloning ht.el...
Building org-super-agenda → Cloning ht.el...done
Building org-super-agenda → Building ht...
Building org-super-agenda → Building ht...done
Building org-super-agenda → Cloning ts.el...
Building org-super-agenda → Cloning ts.el...done
Building org-super-agenda → Building ts...
Building org-super-agenda → Building ts...done
Building org-super-agenda...
Building org-super-agenda...done

Second Run Output
Test run with version: prerelease (HEAD -> develop, origin/develop) 949a0a7 2020-12-13
Building org-plus-contrib...
Building org-plus-contrib...done
Building org...
Building org...done

latest develop branch is byte compiling in a separate process and has the altered Org mode recipes.
Hoping that takes care of it. Appreciate any testing/feedback.

latest develop branch is byte compiling in a separate process and has the altered Org mode recipes.
Hoping that takes care of it. Appreciate any testing/feedback.

This appears to work for me. I had to update my custom org-plus-contrib recipe to use the new built-in one, which appears to work well. Thank you!

This appears to work for me. I had to update my custom org-plus-contrib recipe to use the new built-in one, which appears to work well. Thank you!

I'll gladly test later. What's the recommended use-package/straight-use
form for this now?
On Sat, 2 Jan 2021 at 07:54, Aaron Jensen<EMAIL_ADDRESS>wrote:

This appears to work for me. I had to update my custom org-plus-contrib
recipe to use the new built-in one, which appears to work well. Thank you!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raxod502/straight.el/issues/624#issuecomment-753440788,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYTC2OEXBNFVV6OWKHP2TSX27MBANCNFSM4TGXVASQ
.

I'll gladly test later. What's the recommended use-package/straight-use
form for this now?
On Sat, 2 Jan 2021 at 07:54, Aaron Jensen<EMAIL_ADDRESS>wrote:

This appears to work for me. I had to update my custom org-plus-contrib
recipe to use the new built-in one, which appears to work well. Thank you!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raxod502/straight.el/issues/624#issuecomment-753440788,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYTC2OEXBNFVV6OWKHP2TSX27MBANCNFSM4TGXVASQ
.

I just switched to (use-package org-plus-contrib) and other packages too :after org-plus-contrib and it seems to work fine.

I just switched to (use-package org-plus-contrib) and other packages too :after org-plus-contrib and it seems to work fine.

but org-plus-contrib takes care of loading org? What a minefield. I've kind
of given up trying to understand it;)
On Sat, 2 Jan 2021 at 12:14, Adam Spiers<EMAIL_ADDRESS>wrote:

I just switched to (use-package org-plus-contrib) and other packages to :after
org-plus-contrib and it seems to work fine.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raxod502/straight.el/issues/624#issuecomment-753461156,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYTCZRXSPEUNRMBGZPKODSX353BANCNFSM4TGXVASQ
.

but org-plus-contrib takes care of loading org? What a minefield. I've kind
of given up trying to understand it;)
On Sat, 2 Jan 2021 at 12:14, Adam Spiers<EMAIL_ADDRESS>wrote:

I just switched to (use-package org-plus-contrib) and other packages to :after
org-plus-contrib and it seems to work fine.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raxod502/straight.el/issues/624#issuecomment-753461156,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYTCZRXSPEUNRMBGZPKODSX353BANCNFSM4TGXVASQ
.

If you look at the latest source, you'll see that straight.el translates both org and org-plus-contrib into recipes which start with (org ...) and both have "lisp/*.el" in their :files property, but only org-plus-contrib also has "contrib/lisp/*.el" in its :files property.

If you look at the latest source, you'll see that straight.el translates both org and org-plus-contrib into recipes which start with (org ...) and both have "lisp/*.el" in their :files property, but only org-plus-contrib also has "contrib/lisp/*.el" in its :files property.

I've no idea what that means or the consequences. I'll just use-package
org-plus-contrib and see how it goes. Many thanks.
On Sat, 2 Jan 2021, 14:22 Adam Spiers<EMAIL_ADDRESS>wrote:

If you look at the latest source, you'll see that straight.el translates
both org and org-plus-contrib into recipes which start with (org ...) and
both have "lisp/.el" in their :files property, but only org-plus-contrib
also has "contrib/lisp/.el" in its :files property.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raxod502/straight.el/issues/624#issuecomment-753473136,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYTC7O33ZK5KBVUMLK2P3SX4MZVANCNFSM4TGXVASQ
.

I've no idea what that means or the consequences. I'll just use-package
org-plus-contrib and see how it goes. Many thanks.
On Sat, 2 Jan 2021, 14:22 Adam Spiers<EMAIL_ADDRESS>wrote:

If you look at the latest source, you'll see that straight.el translates
both org and org-plus-contrib into recipes which start with (org ...) and
both have "lisp/.el" in their :files property, but only org-plus-contrib
also has "contrib/lisp/.el" in its :files property.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raxod502/straight.el/issues/624#issuecomment-753473136,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYTC7O33ZK5KBVUMLK2P3SX4MZVANCNFSM4TGXVASQ
.

Thank you everyone for you patience and testing efforts.
Glad to see things are working. I'll consider this bug closed for now. If the issue crops up again for anyone, feel free to leave a comment and we can reopen it.
@reilyrg: I understand the confusion around Org vs "Org plus contrib". In Org 9.5, the plan (as I understand it) is to break out the contrib packages into their own separate repository.
This should simplify things and hopefully clear up some of the confusion. If you're interested in the details:
<EMAIL_ADDRESS>

Thank you everyone for you patience and testing efforts.
Glad to see things are working. I'll consider this bug closed for now. If the issue crops up again for anyone, feel free to leave a comment and we can reopen it.
@reilyrg: I understand the confusion around Org vs "Org plus contrib". In Org 9.5, the plan (as I understand it) is to break out the contrib packages into their own separate repository.
This should simplify things and hopefully clear up some of the confusion. If you're interested in the details:
<EMAIL_ADDRESS>

Thank goodness. org-plus-contrib is quite frankly bizarre. Unfortunately, in the absence of some upstream packaging changes, I do not think it is possible to smooth over the unintuitive behavior in straight.el (really, org-plus-contrib should be a virtual package consisting of "org" and "org-contrib" separately, but there's no concept of virtual packages or anything like that).
@progfolio One thing I notice with the new recipes is that it is now possible for (straight--convert-recipe 'org-plus-contrib) to return a recipe whose package is not named org-plus-contrib -- I believe such a situation never happened before. I don't know if that would cause any problems, and clearly your implementation is working well enough so far, but I wanted to call your attention to this in case something does crop up later.

Thank goodness. org-plus-contrib is quite frankly bizarre. Unfortunately, in the absence of some upstream packaging changes, I do not think it is possible to smooth over the unintuitive behavior in straight.el (really, org-plus-contrib should be a virtual package consisting of "org" and "org-contrib" separately, but there's no concept of virtual packages or anything like that).
@progfolio One thing I notice with the new recipes is that it is now possible for (straight--convert-recipe 'org-plus-contrib) to return a recipe whose package is not named org-plus-contrib -- I believe such a situation never happened before. I don't know if that would cause any problems, and clearly your implementation is working well enough so far, but I wanted to call your attention to this in case something does crop up later.

I don't know if it's something particular to my setup, but I still frequently have to rebuild org.

I don't know if it's something particular to my setup, but I still frequently have to rebuild org.

I tracked this down. Because I launch Emacs on a mac in different ways, invocation-directory and invocation-name can vary. When they vary, org rebuilds because of:
https://github.com/raxod502/straight.el/blob/e98ea33388196bfe30c7a69e2abd5a93a8a701f8/straight.el#L2820
Instead, I hardcoded the recipe with EMACS=emacs which is sufficient for me and it no longer rebuilds unnecessarily
(straight-use-package
 `(org
   :type git
   :repo "https://code.orgmode.org/bzg/org-mode.git"
   :depth full
   :build (:not autoloads)
   :local-repo "org"
   :pre-build ,(list
                (concat (when (eq system-type 'berkeley-unix) "g")
                        "make")
                "autoloads"
                "EMACS=emacs")
   :files (:defaults "lisp/*.el"
                     "contrib/lisp/*.el")))

I tracked this down. Because I launch Emacs on a mac in different ways, invocation-directory and invocation-name can vary. When they vary, org rebuilds because of:
https://github.com/raxod502/straight.el/blob/e98ea33388196bfe30c7a69e2abd5a93a8a701f8/straight.el#L2820
Instead, I hardcoded the recipe with EMACS=emacs which is sufficient for me and it no longer rebuilds unnecessarily
(straight-use-package
 `(org
   :type git
   :repo "https://code.orgmode.org/bzg/org-mode.git"
   :depth full
   :build (:not autoloads)
   :local-repo "org"
   :pre-build ,(list
                (concat (when (eq system-type 'berkeley-unix) "g")
                        "make")
                "autoloads"
                "EMACS=emacs")
   :files (:defaults "lisp/*.el"
                     "contrib/lisp/*.el")))

Maybe those variables should be evaluated at build time, rather than at recipe expansion time? I can't think of a good reason we'd want to rebuild Org just because a different path to Emacs is used. Perhaps if it's a different Emacs version, but in that case the build cache will be invalidated anyway.

Maybe those variables should be evaluated at build time, rather than at recipe expansion time? I can't think of a good reason we'd want to rebuild Org just because a different path to Emacs is used. Perhaps if it's a different Emacs version, but in that case the build cache will be invalidated anyway.

Agreed. I'll look into it.

Agreed. I'll look into it.


UTable : Add the ability to separate sort column and sort direction

Description
In some context, the fact that the sort ref is an object is struggling. In my case, I would like to use useFetch, I cannot set the query params with a ref so the useFetch loose reactivity :
<template>
 <UTable v-model:sort="sort">
</template>

const sort = ref({
  direction: 'desc',
  column: 'year'
})

// here the change of the sort does trigger the request but keep the initial value as the reactivity is lost in my query params
const {data: rates, error, pending} = await useFetch(
    `/api/rates`,
    {
      method: 'GET',
      query: {
        page: page,
        order_by: sort.value.colum,
        order_direction: sort.value.direction
      },
      watch: [page, sort],
    }
)

What could be nice :
<template>
 <UTable v-model:sort-column="sortColumn"  v-model:sort-direction="sortDirection">
</template>

const sort = ref({
  direction: 'desc',
  column: 'year'
})

const sortColumn = ref('year')
const sortDirection = ref('desc')

// here the change of the sort does trigger the request but keep the initial value as the reactivity is lost in my query params
const {data: rates, error, pending} = await useFetch(
    `/api/rates`,
    {
      method: 'GET',
      query: {
        page: page,
        order_by: sortColum,
        order_direction: sortDirection
      },
      watch: [page, sortColum, sortDirection],

    }
)

I can do this :

const {data: rates, error, pending} = await useLazyFetch(() => `/api/rates?page=${page.value}&order_by=${sort.value.column}&order_direction=${sort.value.direction}`)

But it becomes really unreadable with a lot of parameters
Or if anyone have another suggestion
Additional context
No response

To keep the reactivity, you can do something like this:
const query = computed(() => ({ order_by: sort.value.column, order_direction: sort.value.direction }))

const { data: rates, pending } = await useFetch<User[]>('/api/rates', { query, default: () => [] })

Thank you !
Pretty sure I tried with a computed, but I must have made a typo anywhere.
Thanks a lot, it now works, you can close the ticket if you think the initial request doesn't make sense 👍


Compile all provided files in java

Oh ok I see now why you didn´t do this before

Ok so my idea to fix this would be to rename the files to the correct name of the class inside. I wrote a regex to extract the name but I'm not an expert on that so I'm not sure if it always works
(?<=public[\s\S]*(class|enum|record|interface)[\s\t]*)([^\s\t{]*)

Ok that won´t work due to security issues with that but it should be possible to check if the file already has the correct name and in that case compile using javac. Otherwise it could use the previously used method that only compiles a single class

We had a regex that was used in the previous version of Piston when it was using LXC still. It had similar flaws I think. You may wanna check out some of the other JVM-based langs like groovy or scala for some ideas too, but none are perfect imo.
Also, if you get it working, make sure to add it to a separate compile script like in other packages (checkout the contributor's guide)

Closing due to inactivity. Feel free to reopen.


understand and rationalize com link usage

I took a look at what Bohemia was doing here and they're just setting properties on the IFrame window directly instead of using a postMessage wrapper.  Some quick experimentation suggests this would work for us as well (and I think is a bit easier to conceptualize over something like comlink assuming we expose methods on the window at epected points along iframe creation/init)

agreed. like i've mentioned before i'd like to see this generalized so that it's not couples to browser and post message and we can use it for cross process in node, etc.


fix: #417 landing page avatar displaying bug

Sure! I'll try to deal with it.

Hi @MrOrz  I just update the branch.
BTW, it's my first time using Apollo, so I'm not really sure if it's an expected practice.🤧
Please advise as necessary, thank you!


Update version of php, remove mongo dependency

I've added an apt repo in order to upgrade from php 5.5 to 5.6, since several packages don't support 5.5.
Also, the doctrine/mongodb seemed superfluous as not used (and composer didn't seem to recognized that extension was already installed, thus failing install)

Hi @jon-acker, I'm closing this in favor of https://github.com/qandidate-labs/broadway-demo/pull/15.


Scheduling task in "effectively infinite" future fails not nicely

Expected behavior
Some nice way of failing would be nice, like "Don't schedule tasks into Int.max nanoseconds into the future", that's a bad idea"! Or whichever the limit inherent to kevent is.
Likely a fatalError is fine for this, however perhaps provide a better error message for it than currently.
Actual behavior
Scheduling into effectively infinite future crashes not nicely, in kevent as invalid argument:
Fatal error: unexpected error while executing EventLoop kevent(kq:changelist:nchanges:eventlist:nevents:timeout:) failed: Invalid argument (errno: 22): file /Users/ktoso/code/swift-nio/Sources/NIO/EventLoop.swift, line 1041
Exited with signal code 4
Steps to reproduce
If possible, minimal yet complete reproducer code (or URL to code)
    public func testScheduleInfinite() throws {
        let nanos: NIODeadline = .now()
        let amount: TimeAmount = .nanoseconds(.max)
        let eventLoopGroup = MultiThreadedEventLoopGroup(numberOfThreads: 1)
        defer {
            XCTAssertNoThrow(try eventLoopGroup.syncShutdownGracefully())
        }
        let value = try eventLoopGroup.next().scheduleTask(in: amount) {
            true
        }.futureResult.wait() // should never complete or rather error nicely
    }

SwiftNIO version/commit hash
master right now; 803d36b7
Swift & OS version (output of swift --version && uname -a)
Any swift version

thanks, we should definitely not hand that to kevent but fail the promise ourselves with a better error or maybe never schedule it.

I've started investigating this. I see a check in kevent_internal() of ts.tv_sec & 0xFFFFFFFF00000000) != 0) this makes sense to me as the callweheel structure (at least in FreeBSD) works on a time value specified in 32.32. Though when I create an event I can only create values < 2^30 seconds (I'll keep investigating to reconcile the two). Either way though this is a kernel restriction on the kevent timeout  due to limitations on data types, which in term is determined by HW clock stability.
I think - at this stage - that what is needed is better error handling for the EINVAL returned from the syscall.

Resolved in #1642.


Inconsistencies among browsers on page-content padding when toolbar is in

This is a (multiple allowed):

[x] bug

[ ] enhancement

[ ] feature-discussion (RFC)

Framework7 Version: 2.0.7

Framework7-Vue Version: 2.0.10

Platform and Target: Browsers (both mobile and desktop) and IE

What you did
I've created the following template: (the form encapsulates all tabs because I want the form changed event so I can persist my vue-model to localStorage (I need my model, not only the form)):
<template>
    <f7-page name="talents-curiculums-create" :page-content="false">
        <form id="curriculumForm" class="form-store-data extra-bottom-margin" style="height:100%;" @form:storedata="storeData">
            <f7-tabs animated>
                <f7-tab id="tab0" class="page-content">

...

                </f7-tab>
            </f7-tabs>
        </form>

        <f7-toolbar>
            <select class="jumpList" v-model="selectedTab">
                <option value="0" :selected="selectedTab === 0">Introdução</option>
...
            </select>
        </f7-toolbar>
    </f7-page>
</template>

Expected Behavior
Chrome behavior:

Actual Behavior
FireFox Quantum, Edge and Internet Exploder 11:

BUG: the bottom padding when a page has a toolbar on ios theme just works on Chrome.
I assume that works as well on mobile and Safari, but I'm not sure (and, yes, I know it was not supposed to work in non mobile browsers but, I just glad it does 99.9% perfectly)

Yeah, it is a browser's issue with padding on overflow: auto containers. Check the simplest fiddle https://jsfiddle.net/Lcuje10p/
But, as the main target of F7 is Safari(webkit) and Chrome(blink) where it is working as it is expected, we are fine with it. Otherwise, some hacks must be added

As a workaround for such browsers only this can be added:
.page-content:after{
  content: '';
  height: 48px; /* required padding bottom */
  display: block;
  width: 1px;
}


Please support 'with' block access with '[key]' syntax

Current with block only support .key and \key. It is kind of useful but is limited only to the const key.
I would love if YueScript make 'with' support variable key [key]. It would very helpful to reduce typing variable names.
Thanks and regards!

Got feature added.
with tb
  [1] = [2]\func!
  print [3]
  with [abc]
    [4] = 1
  [] = "abc"

now compile to
local _with_0 = tb
_with_0[1] = _with_0[2]:func()
print(_with_0[3])
do
  local _with_1 = _with_0[abc]
  _with_1[4] = 1
end
_with_0[#_with_0 + 1] = "abc"

Done!
Thanks you very much!


fix: set default format in table date type cell

Description

set default format YYYY-MM-DD HH:mm in Orignal Date Format and Display Date Format in table widget cell type Date.
add No Selection additional option in date format

In DropDownControl add support for set defaultValue.

Fixes # (#3318)
Type of change

Please delete options that are not relevant.

Bugfix (non-breaking change which fixes an issue)
Breaking change (fix that would cause existing functionality to not work as expected)

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ]  I have commented my code, particularly in hard-to-understand areas
[ ]  I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ]  I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

Test coverage results :test_tube:
:red_circle: Total coverage has decreased
// Code coverage diff between base branch:release and head branch: fix/dropdown-default-value

Status
File
% Stmts
% Branch
% Funcs
% Lines

:red_circle:
total
48.93 (-0.01)
28.73 (-0.01)
26.77 (-0.01)
49.31 (-0.01)

:red_circle:
app/client/src/components/propertyControls/DropDownControl.tsx
38.46 (-5.02)
0 (0)
28.57 (-4.76)
39.13 (-5.87)

/ok-to-test sha=9b91a09

Tested and Works on deploy preview.

@SatishGandham kindly review

@somangshu , What does "No result" mean? Shouldn't that be no selection or please select

@SatishGandham that makes sense. Can you point out where do you see this? On the deploy preview, I see it as no selection


[Feat]: adding light mode / dark mode option

Kindly assign this to me.I am eagerly waiting to contribute to this project.

what's the update? @souvik-maity

Kindly give me 2 more days time as my end semester exams are going on.I will complete it by Tuesday. I hope that you understand and give a positive response.

I would like to add a light mode/dark mode option with an attractive icon. Could you please consider assigning this issue to me? I'm committed to dedicating my time and effort to delivering a high-quality design that aligns with the project's goals.
Looking forward to the opportunity to work on this exciting project!

@Anushkabh I am getting an error when I am trying to clone the forked repository. I have tried it multiple times but still the issue is there. Kindly look into it ,I shall be grateful for your help.

@souvik-maity can you please mention your Pull Request here?


serializers: support refs

old code:
    def process_metadata(self, record, response_ctx, *args, **kwargs):
        return (
            copy.deepcopy(record.replace_refs())
            if self.replace_refs
            else record.dumps()
        )

This is being handled in inveniosoftware/product-rdm#11 and won't use JSONRef so I'm closing issue until we start on the milestone.


Can't  get Information about the authenticating user: Not Authorized

Hi everyone,
JTumblr failled to get Information about the authenticating user! com.tumblr.jumblr.exceptions.JumblrException: Not Authorized
at com.tumblr.jumblr.JumblrClient.user(JumblrClient.java:83)
Thanks in advance.

Are you authenticating prior to making the API call(s)? If so can you provide the sample code minus your security credentials?

@KevinTCoughlin yes I'm authenticating prior to making the API requests! This morning everything worked well but after about two hours it failled again when I tried to get my following list ! I think it is something related to API calls limit !!

Yea that's possible. Have you looked at the x_ratelimit_* headers returned?

No I never heard about it but I will absolutely check this out ! can you give me some further information! Thanks @KevinTCoughlin  I really appreciate your help!

Hii @KevinTCoughlin
I dont understand ! I get my blog info but I cant get my following list! it keeps returning this:
Exception in thread "AWT-EventQueue-0" com.tumblr.jumblr.exceptions.JumblrException: Not Authorized
at com.tumblr.jumblr.request.RequestBuilder.clear(RequestBuilder.java:151)
at com.tumblr.jumblr.request.RequestBuilder.get(RequestBuilder.java:92)
at com.tumblr.jumblr.JumblrClient.user(JumblrClient.java:83)
at com.tumblr.mine.MyTumblrManager.TumblrManager.getMyFollowing(TumblrManager.java:784)
at com.tumblr.mine.MyTumblrManager.TumblrManager$5.actionPerformed(TumblrManager.java:648)
at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$500(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
does anyone experiencing this too ? please any help would be appreciated!

@azhwani I actually just had this same issue. Clearing my cookies, regenerating my token in the API console, copying and running the sample code for /user/following returned the correct output.
Can you try the following:

Clear your browser cookies
Go to https://api.tumblr.com/console/calls/user/following for your app that is authenticated
Copy the Java sample code (it will have your new generated tokens)
Run it, it should work now

Here's my working code with the credentials removed:
final JumblrClient client = new JumblrClient("<token>", "<secret>");
client.setToken("<token>", "<secret>");

try {
    final List<Blog> blogs = client.userFollowing();
    blogs.size();
    // 20
} catch(JumblrException e) {
    System.out.println(e.getMessage());
}

Hi @KevinTCoughlin
I tried everything u said but I I still having same issue ! i run you sample code with new generated tokens and it does not work!

Are you using the correct tokens when instantiating your Jumblr client? I successfully got my following list using the above code when I used tokens that I had authorized for my blog.
final JumblrClient client = new JumblrClient("consumer_key", "consumer_secret");
client.setToken("oauth_token", "oauth_token_secret");

Make sure your consumer keys are first, then OAuth tokens for the blog you authorized in the following setToken call. I'm fairly certain this is an auth issue on your end because I was able to get it to work.

Closing this issue as I was unable to reproduce the error and could correct retrieve my authenticated blog's following list. @azhwani if this continues to happen I suggest opening a thread in the Tumblr API Google Group.

@KevinTCoughlin Thank u so mutch for ur effort and replies everything works perfectly now :) Thanks again !

No problem, thanks for the follow up.


Unable to download sample_weights due to repository being over it's data quota.

I got an error message:

This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

Could external download links for these files be provided?
Edit:
I found this google drive link in issue #59
https://drive.google.com/drive/folders/1cJLPgGfEuFAQzBKbXQtSGXxLXssw1D9f?usp=sharing
I recommend adding this link to the README.md, so that future adventurers can be spared from spending 2h figuring out what LFS is and how to download such files, just to nearly give up, and grasping at straws, finally dig out the regular download link in an old closed thread.
Edit2:
Much lighter RRDN model, trained with GANs, mentioned in this comment has a broken link, and doesn't seem to be included in the google drive either. Does anyone have a working download link to it?

Hi,
you're not supposed to use git-lfs any longer. There is an API for model weights now, look at the general readme or the weights readme or the notebook tutorials.
Did you find the info for it on the repo somewhere? If so, please point it out so I can remove it. Thank you!

I managed to somehow miss the notebooks folder, which was my fault seeing that it is mentioned in the main readme. Notebooks folder was very useful and helped me to get everything working.
rrdn-C4-D3-G32-G032-T10-x4_epoch299 model is insane, the results are too good to be true, yet somehow still true. Thank you for help and for all your hard work!


show collab objkts under "Creations" tab of the core collaborator IF there is only one core collaborator, and the work is signed

Currently, it's possible to create a collab contract where you are the only core collaborator, with others as beneficiaries. This is a powerful way to use Teia for:

automatic donations to charity (eg. Ukraine)
automatic donations to Breadfond
automatic payments to a service (like conjure, where I send 20% of what I make from art I made with their tool)
having a nice contract based collection of art that can be viewed in one place

an example of a collab setup like this can be seen here: https://teia.art/collab/ideocartography_realized_retired_20220711
[EDIT: updated link]
Given that the art in this type of "collab" contract really does only have one creator, I believe that it makes sense to list these under the "Creations" tab of that core collaborator, rather than the "Collabs" tab. I also hope (but don't know for sure) that this could also improve discoverability for these pieces.

i would really like this feature as well!


Support for TPL & async await

I would like to use your library to add support for Consul to Microsoft Orleans (https://github.com/dotnet/orleans) but the lack of async support is a big issue for that project.
I see there is a PR which has switched the client code to use HttpClient, do you intend to build on this and move to supporting async requests?
Thanks for the work so far!

Do you mean make the rest of the calls async (e.g. Client.KV.Put)? The reason that most of them are sync is because A. when I wrote it the first time, it was a heck of a lot easier to set up that way. and B. all the Consul Go API calls are sync, and this was mostly just trying to be a copy of the Go API but in C#.
I'd probably make a separate namespace like Consul.Async unless you have a better idea - I don't want to modify the existing signatures to be async, since that breaks every downstream consumer of this package, and I don't want to add *Async (e.g. Client.KV.PutAsync, ListAsync etc.) copies of every method and bloat the main interface.
The major reason I re-wrote the client to use HttpClient is someone wanted DNXCore50 support and WebRequest/WebResponse doesn't exist in .NET Core.
I'm very much open to discussion about this, so please let me know what exactly you mean by "async support".

Also FYI, creating an async client should be trivial. It's literally just copying the normal client interfaces, adding the async keyword to each method and replacing the call to Execute() in each method with ExecuteAsync(). I'll give it a shot today.

Thanks @highlyunavailable that's great news.
To clarify I did mean using the *Async methods on HttpClient, adding the async/await keywords and returning Task or Task from all async methods as per this example https://msdn.microsoft.com/en-us/library/hh191443.aspx#Anchor_9
Thanks again.

Yeah, okay. I already sort of do that:
https://github.com/PlayFab/consuldotnet/blob/develop/Consul/Client.cs#L524-L529
Each other call (KV.Put etc.) is using that Execute() method which just waits for the async to complete.
It feels cleaner to me to make a Consul.Async namespace with a ConsulAsyncClient class that implements async from the get-go and then chain new copies of the methods off of it rather than try to hang Sync and Async methods off of the same class, plus it means I have interface freedom for the new class.
Unless you have any strong objections to this method, the above is what I'm going to do.

Well, I found one strong objection: it's a huge pain to do it that way. I see why we have 5 billion DoodlyDoAsync methods in the TPL etc. It probably does make more sense to make an interface to just hang these off of and then implement it on the existing classes.

"It probably does make more sense to make an interface to just hang these off of and then implement it on the existing classes."
I agree.

I'm just trying to figure out a good way to do this - the problem is such:
https://github.com/PlayFab/consuldotnet/blob/master/Consul/ACL.cs#L336-L360
Note how in that chunk of code, ACL is of type IACLEndpoint.
Unless I want to expand the existing interface IACLEndpoint to include CreateAsync etc. which I would rather not (since I know people write unit tests against a mocked version of that interface), I'm going to need some other way to attach the class to the Client object.
This is kind of why I wanted a class specifically for async, since I've sort of painted myself into a corner with the interfaces - either I change the interface, or I have to expose this functionality some other way.

I may have misunderstood the problem, but it doesn't look like that code is making any IO bound calls so it doesn't need an Async version?
Also it is best where possible to use the await keyword rather than GetAwaiter().GetResult().  There is a good explanation of why here: http://blog.stephencleary.com/2014/12/a-tour-of-task-part-6-results.html
Apologies but I will be AFK from now for the evening.

Yeah, I'll figure it out. I'll have something ready for you to take a look at to see if it's something you'd like to use by the time you come back.

Why not go more radical - only expose async API, and document how to call them synchronously?

@sergeybykov because this library was originally created as a sync library, so either I make an Async version or I break compatibility for everyone that was using it before I rewrote it to be all-Async.
It's not a bad idea though, and it would let me clean up some of the uglier interfaces, and since old versions are still in nuget, it won't really break people's code unless they want to update their copy of the library to use new Consul functionality.
What is the best way to call an async API synchronously anyway? I always hear "don't use GetAwaiter().GetResult() but that seems like the best option if your method isn't already async (which means you can't use await). I'm not really a C# developer by trade, so I don't know all the best ways to do TPL stuff.

@highlyunavailable, I see that you need some advice on async stuff, so here goes:
I had the same issues when I implemented my .NET ZooKeeper client. Like you, I wanted to provide both sync and async APIs, but that was a bad idea. So I converted all blocking code to async. Now I was faced with the question of whether I should be using ConfigureAwait(false) in my code. After reading this and this, i made sure every await in my code ended with ConfigureAwait(false). Done, but now how should people call my async methods in a blocking fashion? however they like.
In conclusion : as long as your library is async all the way and every await ends with ConfigureAwait(false) you're supporting both async and sync APIs.

@shayhatsor thanks for the advice. The whole ConfigureAwait(false) thing feels like cargo cult coding to add to EVERY await, but I'm used to more sane Task style libraries (mostly Go) so the entire TPL feels quite jarring to me.
@sergeybykov suggested a way where I provide both sync and async versions (e.g. there will be client.KV.Get() (sync) and client.KVAsync.GetAsync() (async)). Under the hood, the sync versions just call the async versions and block with GetAwaiter().GetResult().
I'd like to rewrite the whole thing to be async but I know most of the current users consume the Sync APIs. Maybe the right answer is just to say "Hey we changed everything by making it async, if you need sync just add GetAwaiter().GetResult() to your code or make your stuff async and await properly. Peace out."

The whole  ConfigureAwait(false)  thing feels like cargo cult coding (despite it being official guidance, of course) to add to EVERY await, but I'm used to more sane Task style libraries (mostly Go) so the entire TPL feels quite jarring to me.

I know... it's annoying. the best part of it is that if you forget it even once, your sync users will get a nice deadlock.

@sergeybykov suggested a way where I provide both sync and async versions (e.g. there will be  client.KV.Get()  (sync) and  client.KVAsync.GetAsync()  (async)). Under the hood, the sync versions just call the async versions and block with  GetAwaiter().GetResult() .

you can do that, it's ugly but will work.

I'd like to rewrite the whole thing to be async but I know most of the current users consume the Sync APIs. Maybe the right answer is just to say "Hey we changed everything by making it async, if you need sync just add  GetAwaiter().GetResult()  to your code or make your stuff async and await properly. Peace out."

Even if you decide to currently provide both sync and async APIs you still must:

convert all your library to async except only the public surface sync methods. this is because having any blocking code called from an async method effectively makes the whole call chain not purely async. don't go there.
use ConfigureAwait(false) everywhere. if you don't, users will get deadlocks when they call your code synchronously from anything that has a synchronization context e.g. GUI or ASP.NET.

Another path to consider is add async methods now, mark sync methods as deprecated (now or later), remove sync methods in a few releases.

I'm going to just make it fully async. The reason is I'm already introducing breaking changes (Client renamed to ConsulClient) in the next release and I'm converting to use the HttpClient classes instead of WebRequest/WebResponse and XUnit from MSTest, so there are already plenty of changes that will cause havoc in other small ways, plus I'm not confident that I won't cause deadlocks with my half-baked use of HttpClient so I may as well do this the right way and just make a clean break between the releases.

Thanks for your help, everyone.
I've got a fully async client here:
https://github.com/PlayFab/consuldotnet/tree/feature/asyncclient
Nuget packages:
https://ci.appveyor.com/project/highlyunavailable/consuldotnet/build/<IP_ADDRESS>-feature/asyncclient/artifacts
Could you take a quick look to see if it's what you need @PaulNorth ?
Note: Somethings are still not going to be "truly" async - for example, Locks and Semaphores use GetAwaiter().GetResult() because you can't await in a lock {} block - same for anything in finally {} blocks. All the primitives (K/V store, Sessions, etc.) are async though.
If it's what you're looking for then I'll update the readme with big warning signs and roll out a release.

Note: Somethings are still not going to be "truly" async - for example, Locks and Semaphores use  GetAwaiter().GetResult()  because you can't await in a  lock {}  block - same for anything in  finally {}  blocks. All the primitives (K/V store, Sessions, etc.) are async though.

@highlyunavailable, there are async versions of those:

semaphore - use SemaphoreSlim.WaitAsync
lock - use AsyncLock.

about awaiting in finally blocks, you should use C# 6. AFAIK this is orthogonal to the .NET version.
if going the C# 6 route isn't an option, you can do this which is a good solution.

Great work everybody! I will pick this up today and include it in the improvements I am making to the Consul-Orleans Membership Provider.

@shayhatsor I mean the Consul kind of (Lock)[https://consul.io/docs/guides/leader-election.html]/(Semaphore)[https://consul.io/docs/guides/semaphore.html] - the distributed ones that use Consul's consistent state to allow multiple machines to contend for a K/V in Consul, not the C# kind.
I noticed that await in finally was working on my machine (VS2015) but not on Appveyor, so it sounds like I need to upgrade the CI box to use C# 6 and that problem will be fixed.
@PaulNorth please note that this isn't released to nuget, I'd just like you to test it and make sure it's what you need and that there are no surprises before I do make a real release. There was plenty of redesign so it's entirely possible that I introduced a new bug somewhere. I'm fairly confident I didn't since all the tests pass, but I'd like to have someone try it at least.

@shayhatsor I mean the Consul kind of Lock/Semaphore - the distributed ones that use Consul's consistent state to allow multiple machines to contend for a K/V in Consul, not the C# kind.

ok, but you noted:

you can't await in a lock {} block

Your code can still be fully async. as an example I've looked at your lock implementation.

Replace all locks on _lock with AsyncLock on _lock.
Keep the regular locks on _heldLock, as these provide freshness to the value. You can also use
Fenced<bool> which provides freshness without acquiring a lock (this is just a super optimization, basically because i'm crazy)

Having switched to the new async build the methods we are using all appear to be working just fine.  Our usage is by no means an exhaustive test; we are only using KW.Put, KW.Get, KW.CAS, Agent.ServiceRegister, Agent.ServiceDeregister & Catalog.Service.
The migration was pretty painless and I can only imagine your existing clients will be grateful to have a purely async interface.
Cheers!

@shayhatsor Okay, fair enough, I can replace much of the lock {} blocks with AsyncLock, it seems to work fine, but I'm also unsure of what you mean by your earlier statement "follow the async all the way rule to the public surface methods." in relation to what you said before: "1) convert all your library to async except only the public surface sync methods. "
Is this saying it's fine if Acquire(), Release() etc are not async and use GetAwaiter().GetResult() as long as all the calls below that in the call chain are async? If so, then this is the case now - it's completely async except for those surface methods Acquire, Release, Destroy. It also seems weird to have lock-style statements be async since they're supposed to be guarding critical sections, so it feels like you want blocking here.
One final thing I'm struggling with is AutoSemaphore (and AutoLock) - they're a pair of classes that are supposed to let you do using (new AutoLock(lockargs)) {something} and it will do an I/O to Consul, get a lock there, do your thing in the using block, and then the Dispose method does another I/O to unlock.
Bad practice due to long constructor/disposer calls, I suppose, but it's in there now, and I don't see a good way of doing async acquire/release in the constructor and disposer without GetAwaiter().GetResult(). I guess I could deprecate the class, I don't know anyone that is using it.
@PaulNorth Sounds good - I'll cut a release and any further modifications (e.g. the discussion above) will be new releases. Thanks for putting in this issue!

I'm also unsure of what you mean by your earlier statement "follow the async all the way rule to the public surface methods." in relation to what you said before: "1) convert all your library to async except only the public surface sync methods. "
Is this saying it's fine if Acquire(), Release() etc are not async and use  GetAwaiter().GetResult()  as long as all the calls below that in the call chain are async?

ok, i'll explain again, remember I wrote this:

convert all your library to async except only the public surface sync methods. this is because having any blocking code called from an async method effectively makes the whole call chain not purely async. don't go there.

GetAwaiter().GetResult() is a blocking call. you don't want to block the thread while you're waiting for network response. just use await instead .

It also seems weird to have lock-style statements be  async  since they're supposed to be guarding critical sections, so it feels like you want blocking here.

it is weird, but it's what you want. note that you can't await in a normal lock because it is owned by the running thread, which you actually release upon await... so it doesn't really make sense. Instead, you need something that isn't thread related that provides mutual exclusion asynchronously. this is what AsyncLock gives you. Keep in mind that AsyncLock isn't reentant and normal lock is. but I don't see it as a requirement here.

One final thing I'm struggling with is AutoSemaphore (and AutoLock) - they're a pair of classes that are supposed to let you do  using (new AutoLock(lockargs)) {something}  and it will do an I/O to Consul, get a lock there, do your thing in the using block, and then the Dispose method does another I/O to unlock.

what you do is this:

use async factory method instead of the constructor. (make the ctor private)
use async dispose method instead of the dispose.
this does mean you can't use using anymore, as c# doesn't have an async using yet.
but you can make your own. I actually did that in zookeeper. and an example of usage is here

just noticed I haven't explained this:

I'm also unsure of what you mean by your earlier statement "follow the async all the way rule to the public surface methods." in relation to what you said before: "1) convert all your library to async except only the public surface sync methods. "

I'll explain by example. let's say you had your sync class:
public class fooAPI
{
    int Foo1()
    {
    //some blocking code goes here
    }
}

and now you're going async, but wish to also keep the sync methods. you'd do this:
public class fooAPI
{
    int Foo1()
    {
      //this is OK because it's a public surface method that only calls the async code.
      //it isn't part of any async flow, it's here just to keep the sync interface.
      return Foo1Async.GetAwaiter().GetResult();
    }

    Task<int> Foo1Async()
    {
       //some async code goes here
    }
}

Awesome, the example is what I was thinking and we're both thinking the same thing.
Okay, then I'm not going to change anything further this release because that's exactly what I'm doing. Only the Lock/Semaphore stuff is sync, EVERYTHING else (the primitives in KV storage, sessions, etc.) are async. I am not putting any syncs in any async flows.
Thanks for all your help and for explaining this, everyone. I'm going to close this issue now by saying that there's a new release that's all Async except for lock/semaphore at https://www.nuget.org/packages/Consul tagged <IP_ADDRESS>. This is the official package, @PaulNorth
I'll open a new issue for explore converting the distributed locks to fully async.

:+1:

Great cross project collaboration! 4 different teams, and the async support added to Consul.Net in 2 days!


Has anyone created a Chatto PDF Message type?

I am developing an iOS healthcare related messaging app that, so far, has text and photo message types.  I would like to add support for PDF (and/or other file kinds) message types.
The App is built using Chatto and Chatto Additions.
I have implemented 1-on-1 and Group Messaging with Text and Photo message types.  The App also utilizes end-to-end encryption via Virgil Security's E3Kit.
I would like to be able to send and receive messages that contain PDF files.  It would be great if other kinds of files could be exchanged as well.
I am looking for an implementation that can be adapted for my App.

You can just add your custom message type like photoCell. Define your bubble, message model, view model, presenter etc. There is no difference with other message types.

Thanks Pavel.  I knew that I can probably use the Chatto Additions Photo message classes to develop a Files message classes.  I would also need to develop file picker classes for selecting PDFs or other files for message content.  I was hoping to avoid all of that if someone else has done it.

@101airborne there is no need to develop file picker. You can add a new input item for selecting files and use UIDocumentMenuViewController.
I used the PhotoMessageCell to present the files preview and just changed the action when tapping the cell to present/download/share the file

@101airborne there is no need to develop file picker. You can add a new input item for selecting files and use UIDocumentMenuViewController.
I used the PhotoMessageCell to present the files preview and just changed the action when tapping the cell to present/download/share the file

Thank you, OrChenOS.  I will try your suggestion next week when I get back from vacation.

@101airborne there is no need to develop file picker. You can add a new input item for selecting files and use UIDocumentMenuViewController.
I used the PhotoMessageCell to present the files preview and just changed the action when tapping the cell to present/download/share the file

OrChenOS, I experimented with your suggestion to use UIDocumentMenuViewController for picking the files, and found that it works well despite the fact that UIDocumentMenuViewController is now deprecated.  But, I am still having a problem integrating it into a new FilesChatInputItem.  There is next to no guidance in the Chatto wikis on how to develop a new ChatInputItem and associated classes and protocols..  The existing examples PhotosChatInputItem and TextChatInputItem are so dissimilar that neither seems to be a useful guide to develop a FilesChatInputItem and the needed associated classes and protocols.
OrChenOS, have you by any chance pushed your version of a FilesChatInputItem and associated classes?


Update to 1.10.0

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

This PR builds on top of https://github.com/conda-forge/flytekit-feedstock/pull/61 (as I don't have rights to update that).
In addition to that one the following changes have been made:

Dependencies have been sorted alphabetically to make them easier to compare to the ones in the original setup.py. I will also follow up on the flytekit side to make sure those are sorted as well
After (1) I've seen that both typing_extensions as well as typing-extensions have been added as a depdenecy. I've checked and both come from https://github.com/conda-forge/typing_extensions-feedstock/; so luckly no typosquatting.
I've removed responses as that's not present within flytekit

@conda-forge-admin, please rerender

LGTM 🎉🚀


Z Loss in CE

🚀 The feature, motivation and pitch
Often used in pretraining of LMs for stabilization, i.e. the recent Chameleon & PaLM.
Alternatives
flash-attn has implementations of abovementioned features, however, does not support fusing with linear head.
Additional context
No response

Legit ask! We have tracked smooth label at https://github.com/linkedin/Liger-Kernel/issues/81. I modify the title for only Z loss to prevent duplication.

@ByronHsu #take
To support z loss, I just need a little add-ons to #198.
I'll work on it after merging label_smoothing PR.